Multiview light-sheet microscopy

ABSTRACT

A method of imaging a live biological specimen includes generating one or more first light sheets; directing the generated one or more first light sheets along respective paths that are parallel with a first illumination axis such that the one or more first light sheets optically interact with at least a portion of the biological specimen in a first image plane; recording, at each of a plurality of first views, images of fluorescence emitted along a first detection axis; generating one or more second light sheets; directing the generated one or more second light sheets along respective paths that are parallel with a second illumination axis such that the one or more second light sheets optically interact with at least a portion of the biological specimen in a second image plane, and recording, at each of a plurality of second views, images of fluorescence emitted along a second detection axis.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.61/888,869, filed Oct. 9, 2013, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The disclosed subject matter relates to live imaging of biologicalspecimens.

BACKGROUND

Understanding the development and function of complex biologicalspecimens relies critically on our ability to record and quantify fastspatio-temporal dynamics on a microscopic scale. Owing to thefundamental trade-off between spatial resolution, temporal resolution,and photo-damage, the practical approach in biological live imaging hasbeen to reduce the observation of large specimens to small functionalsubunits and to study these one at a time.

SUMMARY

In some general aspects, a method of imaging a live biological specimenincludes generating one or more first light sheets; directing thegenerated one or more first light sheets along respective paths that areparallel with a first illumination axis such that the one or more firstlight sheets optically interact with at least a portion of thebiological specimen in a first image plane; and recording, at each of aplurality of first views, images of fluorescence emitted along a firstdetection axis from the biological specimen. The method includesgenerating one or more second light sheets; directing the generated oneor more second light sheets along respective paths that are parallelwith a second illumination axis such that the one or more second lightsheets optically interact with at least a portion of the biologicalspecimen in a second image plane, in which the second illumination axisis not parallel with the first illumination axis; and recording, at eachof a plurality of second views, images of fluorescence emitted along asecond detection axis from the biological specimen.

Implementations can include one or more of the following features. Forexample, the one or more first light sheets can be generated byactivating two first light source units to generate two first lightsheets; and the generated one or more first light sheets can be directedalong respective paths that are parallel with the first illuminationaxis through the biological specimen by directing the generated twofirst light sheets along respective paths that are parallel with eachother but pointing in opposite directions from each other. The generatedtwo first light sheets can spatially and temporally overlap within thebiological specimen along the first image plane. The temporal overlap ofthe generated first light sheets is within a time shift that is lessthan a resolution time that corresponds to a spatial resolution limit ofthe microscope. A spatial displacement in a tracked structure of thebiological specimen during the time shift can be less than the spatialresolution limit of the microscope.

The one or more second light sheets can be generated by activating twosecond light source units to generate two second light sheets; and thegenerated one or more second light sheets can be directed alongrespective paths that are parallel with the second illumination axisthrough the biological specimen by directing the generated two secondlight sheets along respective paths that are parallel with each otherbut pointing in opposite directions from each other. The generated twosecond light sheets can spatially and temporally overlap within thebiological specimen along the second image plane. The temporal overlapof the generated second light sheets can be within a time shift that isless than a resolution time that corresponds to a spatial resolutionlimit of the microscope. The spatial displacement in a tracked structureof the biological specimen during the time shift can be less than thespatial resolution limit of the microscope.

Each of the first views can be perpendicular to the first illuminationaxis and each of the second views can be perpendicular to the secondillumination axis.

Generating the one or more first light sheets can include activating oneor more respective first light source units; and generating the one ormore second light sheets can include activating one or more respectivesecond light source units. The method can include de-activating the oneor more first light source units while activating the one or more secondlight source units; and de-activating the one or more second lightsource units while activating the one or more first light source units.The one or more first light sheets and the one or more second lightsheets can be at the same wavelength.

The method can also include activating the one or more first lightsource units while activating the one or more second light source units.The one or more first light sheets can be at a first wavelength and theone or more second light sheets can be at a second wavelength that isdistinct from the first wavelength. A first type of structure within thebiological specimen can include a first fluorophore that fluoresces inresponse to light of the first wavelength; and a second type ofstructure within the biological specimen can include a secondfluorophore that fluoresces in response to light of the secondwavelength.

The one or more first light sheets and the one or more second lightsheets can be at the same wavelength. Images of fluorescence emittedalong the first detection axis can be recorded by capturing thefluorescence with a first objective and directing the capturedfluorescence from the objective through an aperture placed at a focalplane of the first objective to block out at least a portion ofout-of-focus fluorescence. Images of fluorescence emitted along thesecond detection axis can be recorded by capturing the fluorescence witha second objective and directing the captured fluorescence from thesecond objective through an aperture placed at a focal plane of thesecond objective to block out at least a portion of out-of-focusfluorescence.

Images of fluorescence emitted along the first detection axis can berecorded by capturing the fluorescence with a first objective anddirecting the captured fluorescence from the objective to a first camerathat records the images of the fluorescence; and images of fluorescenceemitted along the second detection axis can be recorded by capturing thefluorescence with a second objective and directing the capturedfluorescence from the second objective to a second camera that recordsthe images of the fluorescence.

Each of the generated first light sheets can have different polarizationstates from each other and each of the generated second light sheets canhave different polarization states from each other.

The method can include, during generation of the one or more first lightsheets, translating one or more of the biological specimen, the one ormore first light sheets, and the first views at which the fluorescenceis recorded relative to each other along a first linear axis that isperpendicular with the first illumination axis by incremental steps sothat the one or more first light sheets optically interact with thebiological specimen along a set of first image planes that spans atleast a portion of the biological specimen; and for each first imageplane, recording, at the first views, fluorescence produced by thebiological specimen. The one or more of the biological specimen, the oneor more first light sheets, and the first recording views can betranslated relative to each other along the first linear axis byincremental steps by maintaining the position of the biologicalspecimen; and translating the one or more first light sheets and thefirst recording views along the first linear axis. The method caninclude, during generation of the one or more second light sheetstranslating one or more of the biological specimen, the one or moresecond light sheets, and the second views at which the fluorescence isrecorded relative to each other along a second linear axis that isperpendicular with the second illumination axis by incremental steps sothat the one or more second light sheets optically interact with thebiological specimen along a set of second image planes that spans atleast a portion of the biological specimen; and for each second imageplane, recording, at the second views, fluorescence produced by thebiological specimen. The method can include creating an image of thebiological specimen by aligning the images of the recorded fluorescenceat the first and second views, and combining the aligned images using amultiview deconvolution algorithm.

The method can include tracking a structure in the biological specimenbased on the recordings, in which tracking the structure includescreating images of the recorded fluorescence at the first and secondviews, and combining the aligned images.

Recording at the first views can include recording along the detectionaxis that is perpendicular to the illumination axis.

A minimal thickness of each of the light sheets taken along itsrespective detection axis can be less than a cross sectional size of astructure within the specimen to be imaged.

The second illumination axis can be perpendicular with the firstillumination axis.

Fluorescence can be emitted along the first detection axis after orwhile the one or more first light sheets optically interact with thebiological specimen; and fluorescence can be emitted along the seconddetection axis after or while the one or more second light sheetsoptically interact with the biological specimen.

The first illumination axis and the second illumination axis canspatially overlap within the biological specimen.

The light sheet can be generated by generating a laser light sheet.

The light sheet can be generated by forming a sheet of light that has aspatial profile that is longer along a first transverse axis than asecond transverse axis that is perpendicular to the first transverseaxis, in which the transverse axes are perpendicular to a direction ofpropagation of the light sheet. The second transverse axis of the one ormore first light sheets can be parallel with the first detection axisand the second transverse axis of the one or more second light sheetscan be parallel with the second detection axis.

Each first light sheet can be generated by scanning a first light beamemitted from a first light source along a direction transverse to thefirst illumination axis to generate the first light sheet; and eachsecond light sheet can be generated by scanning a second light beamemitted from a second light source along a direction transverse to thesecond illumination axis to generate the second light sheet such thatthe first light beam of the first light sheet does not intersect withthe second light beam of the second light sheet within the biologicalspecimen.

The images of fluorescence emitted along the first detection axis fromthe biological specimen can be recorded by recording, at each of theplurality of first views, images of fluorescence emitted fromfluorophores within the biological specimen along the first detectionaxis; and recording, at each of the plurality of second views, images offluorescence emitted along the second detection axis from the biologicalspecimen comprises recording, at each of the plurality of first views,images of fluorescence emitted from fluorophores within the biologicalspecimen along the second detection axis.

The images of the fluorescence produced by the biological specimen canbe recorded by recording one-photon fluorescence produced by thebiological specimen. The one or more first light sheets can be generatedby generating one or more continuous wave first light sheets; andgenerating the one or more second light sheets comprises generating oneor more continuous wave second light sheets.

The images of the fluorescence produced by the biological specimen canbe recorded by recording two-photon fluorescence produced by thebiological specimen. The one or more first light sheets can be generatedby generating one or more pulsed wave first light sheets.

In other general aspects, a microscope system is configured to image alive biological specimen. The microscope system includes a specimenregion configured to receive a biological specimen and two or moreoptical arms. Each optical arm has an optical path that crosses thespecimen region, each optical arm comprising a light source unit and adetection unit, and each optical arm is arranged along a distinct armaxis, with at least two arm axes being not parallel with each other.Each light source unit includes a light source and a set of illuminationoptical devices arranged to produce and direct a light sheet toward thespecimen region along a respective illumination axis, and a set ofactuators coupled to one or more illumination optical devices. Eachdetection unit includes a camera and a set of detection optical devicesarranged to collect and record images of fluorescence emitted from abiological specimen received in the specimen region along a respectivedetection axis that is perpendicular to one or more of the illuminationaxes, and a set of actuators coupled to one or more of the camera andthe detection optical devices. Each optical arm includes an optical dataseparation apparatus on the path between the specimen region and thelight source unit and the detection unit and configured to separateoptical data between the light source unit and the detection unit.

Implementations can include one or more of the following features. Forexample, the microscope system can include a translation system coupledto one or more of the specimen region, the light source units, and thedetection units, and configured to translate one or more of the specimenregion, the light sheets within the light source units, and thedetection units relative to each other along a linear axis withoutrotating a biological specimen received in the specimen region.

The microscope system can include a specimen holder on which thebiological specimen is mounted so as to be located in the specimenregion.

At least one component can be shared between the light source unit andthe detection unit of each optical arm. The at least one sharedcomponent can include a microscope objective. The at least one sharedcomponent can be placed between the optical data separation apparatusand the specimen region. The microscope objective in each optical armcan have the same focal plane as the other microscope objectives in theother optical arms. The microscope system can include an aperture placedbetween the microscope objective in each optical arm and the camera.

A field of view of the microscope objective in each detection unit canbe at least as great as the diffraction limit of the microscopeobjective.

The microscope objective in each detection unit can have a numericalaperture sufficient to resolve a structure of the biological specimentracked by the detection unit.

The at least one shared component can include a tube lens.

The at least two arm axes that are not parallel with each other can beperpendicular to each other.

The two or more optical arms can include four optical arms, each opticalarm axis being perpendicular to at least two of the other optical armaxes.

The optical data separation apparatus can include a polychroic opticalelement between the at least one shared component and the remainingcomponents of the light source unit and the detection unit. Within eachoptical arm, the light sheet from the light source can be reflected fromthe polychroic optical element, and the fluorescence from the biologicalspecimen can be transmitted through the dichroic optical element.

In other general aspects, a method of imaging a live biological specimenincludes generating a first light sheet; directing the generated firstlight sheet along a path that is parallel with a first illumination axissuch that the first light sheet optically interacts with at least aportion of the biological specimen in a first image plane; recording, ateach of a plurality of first views, images of fluorescence emitted alonga first detection axis from the biological specimen during the opticalinteraction between the first light sheet and the biological specimen;generating a second light sheet; directing the generated second lightsheet along a path that is parallel with a second illumination axis suchthat the second light sheet optically interacts with at least a portionof the biological specimen in a second image plane, in which the secondillumination axis is not parallel with the first illumination axis; andrecording, at each of a plurality of second views, images offluorescence emitted along a second detection axis from the biologicalspecimen during the optical interaction between the second light sheetand the biological specimen.

In other general aspects, a method of imaging a live biological specimenincludes generating a plurality of light sheets; directing the pluralityof light sheets along an illumination axis through the biologicalspecimen such that the light sheets spatially and temporally overlapwithin the biological specimen along an image plane, and opticallyinteract with the biological specimen within the image plane; recording,at each of a plurality of first views, images of the fluorescenceemitted along a first detection axis from the biological specimen due tothe optical interaction between the light sheets and the biologicalspecimen; rotating an orientation of the light sheets; and recording, ateach of a plurality of second views, images of the fluorescence emittedalong a second detection axis from the biological specimen due to theoptical interaction between the rotated light sheets and the biologicalspecimen. The temporal overlap is within a time shift that is less thana resolution time that corresponds to a spatial resolution limit of themicroscope.

In other general aspects, a microscope system for imaging of a livebiological specimen includes a specimen holder on which the biologicalspecimen is mounted; a plurality of illumination subsystems; a pluralityof first detection subsystems; a plurality of second detectionsubsystems; a first translation system coupled to one or more of thespecimen holder, the plurality of illumination subsystems, and theplurality of first detection subsystems, and configured to translate oneor more of the biological specimen, the plurality of light sheets, andthe plurality of first detection subsystems relative to each other alonga first linear axis without rotating the biological specimen; and asecond translation system coupled to one or more of the specimen holder,the plurality of illumination subsystems, and the plurality of seconddetection subsystems, and configured to translate one or more of thebiological specimen, the plurality of light sheets, and the plurality ofsecond detection subsystems relative to each other along a second linearaxis without rotating the biological specimen. Each illuminationsubsystem includes a light source and a set of illumination opticaldevices arranged to produce and direct a light sheet toward thebiological specimen along an illumination axis, and a set of actuatorscoupled to one or more illumination optical devices. Each firstdetection subsystem includes a camera and a set of detection opticaldevices arranged to collect and record images of fluorescence emittedfrom the biological specimen along a first detection axis that isperpendicular to the illumination axis, and a set of actuators coupledto one or more of the camera and the detection optical devices. Eachsecond detection subsystem includes a camera and a set of detectionoptical devices arranged to collect and record images of fluorescenceemitted from the biological specimen along a second detection axis thatis perpendicular to the illumination axis, and a set of actuatorscoupled to one or more of the camera and the detection optical devices.

Implementations can include one or more of the following features. Forexample, microscope system can include a control system connected to theplurality of illumination subsystems, the plurality of first detectionsubsystems, the plurality of second detection subsystems, and the firstand second translation systems, and configured to: send signals to thefirst translation system to change a relative position between thespecimen holder, the plurality of illumination subsystems, and theplurality of first detection subsystems along the first linear axis thatis parallel with a normal to a first set of image planes; for each imageplane in the first set, send signals to the plurality of illuminationsubsystems to cause the light sheets from each of the plurality ofillumination subsystems to spatially and temporally overlap within thebiological specimen along the image plane to thereby optically interactwith the biological specimen; and for each image plane, receive signalsfrom the plurality of first detection subsystems acquiring thefluorescence emitted from the biological specimen. The temporal overlapis within a time shift that is less than a resolution time thatcorresponds to a spatial resolution of the microscope system.

The microscope system provides a useful tool for the in vivo study ofbiological structure and function at a level that captures an entirecomplex biological specimen. It includes a light sheet microscopytechnique that is based on illuminating the biological specimen orsample with a thin sheet of laser light that is directed along a lightaxis, and recording the fluorescence emitted from this thin volumeorthogonally to the light axis. The laser light is thin compared withthe size of the biological specimen. For example, the thin volume couldbe about 100 nm to about 10 μm wide in the direction of the light axis.

Only the in-focus part of the biological specimen is exposed to laserlight, which provides optical sectioning and substantially reducesphoto-damage. Moreover, the fluorescence signal emitted from thein-focus section is detected simultaneously in time for the entirefield-of-view, which provides exceptionally high imaging speeds. Incomparison to confocal microscopy, a commonly used optical sectioningtechnique, imaging speed, signal-to-noise ratio and photo-bleachingrates are improved by up to several orders of magnitude. The microscopesystem may allow further improvement in spatial and temporal resolution,as well as in the conceptual design and complexity of live imagingexperiments.

The microscope system is able to penetrate more than several tens tohundreds of microns into living tissue, thus enabling systems-levelimaging, which is imaging of complex biological specimens or organisms.The microscope system and related process achieve high imaging speeds inthe order of 100 frames per second such that dynamic biologicalprocesses within the complex specimen can be captured at highspatio-temporal resolution.

The microscope system is fast enough to capture fast processes in livespecimens. For example, in live multi-cellular organisms, fastdevelopmental processes can occur between the sequential multiviewacquisitions that occur in sequential multiview imaging systems, andthis precludes accurate image fusion of the acquired data. Themicroscope system therefore avoids spatio-temporal artifacts that canfundamentally constrain quantitative analyses, such as thereconstruction of cell tracks and cell morphologies.

For global measurements of the dynamic behavior and structural changesof all cells in a complex developing biological specimen or organism,data acquisition occurs at speeds that match the time-scales of thefastest processes of interest and with minimal time shifts betweencomplementary views.

The technology framework for simultaneous multiview imaging withone-photon or multi-photon light sheet-based fluorescence excitation isdesigned to deliver exceptionally high imaging speeds and physicalcoverage while minimizing or reducing photo-bleaching and photo-toxiceffects. The microscope system excels at the quantitative in vivoimaging of large biological specimens in their entirety, over longperiods of time and with excellent spatio-temporal resolution.

Similar to its implementation with two-photon excitation, the conceptcan also be combined with Bessel plane illumination, scanned lightsheet-based structured illumination or functional imaging approaches,among others. Thus, the “light sheet” used in the illuminationsubsystems encompasses these other illumination approaches. And, thus,the light sheet could be a scanned light sheet or a static light sheet.The use of laser scanning to generate a light sheet (that is,illuminating only one line in the specimen at a time) enables the use oftwo-photon excitation and Bessel beam illumination, and the temporalcharacter of the scanning process also allows the construction ofcomplex illumination patterns such as structured light sheets (stripesof light).

The imaging technique presented here opens the door to high-throughputhigh-content screening, fast functional imaging and comprehensivequantitative analyses of cellular dynamics in entire developingorganisms. By combining this method with advanced computational toolsfor automated image segmentation and cell tracking, the reconstructionof high-quality cell lineage trees, comprehensive mapping of geneexpression dynamics, automated cellular phenotyping and biophysicalanalyses of cell shape changes and cellular forces are within reach,even for very complex biological specimens.

In comparison to single-view imaging, simultaneous multiview imagingimproves coverage almost four-fold and reveals cellular dynamics in theentire early embryo. Moreover, in comparison to sequential multiviewimaging, simultaneous multiview imaging eliminates or greatly reducestemporal and spatial fusion artifacts during fast nuclear movements,improves temporal resolution, and provides quantitative data forsubsequent computational image analysis, such as automated celltracking. Simultaneous multiview imaging also reduces the energy load onthe specimen by avoiding the redundant iterative acquisition ofoverlapping regions required for sequential multiview imageregistration.

Besides the elimination of spatial and temporal artifacts, simultaneousmultiview imaging also provides excellent temporal resolution. Thispoint can be illustrated with basic parameters quantifying cellulardynamics in the Drosophila embryo, as discussed below. For example, fastmovements of thousands of nuclei occur during the mitotic cycles in thesyncytial blastoderm: In the 12^(th) mitotic cycle, the average movementspeed of dividing nuclei is 8.12±2.59 μm/min (mean±s.d., n=2,798, Huberrobust estimator) and the average nearest neighbor distance is 7.57±1.34μm (mean±s.d., n=1.44×10⁵, Huber robust estimator). Importantly, nearestneighbors are usually not daughter nuclei from the same mother nucleus.Hence, in order to obtain quantitative data for the analysis of nucleardynamics in the entire blastoderm, simultaneous multiview imaging isrequired at an overall speed that ensures, on average, nuclear movementsof no more than half of the nearest neighbor distance between subsequenttime points. Simultaneous multiview imaging of the entire embryotherefore has to be performed with at most about 30 second temporalsampling.

DESCRIPTION OF DRAWINGS

FIG. 1A is a block diagram of an exemplary microscope system that useslight sheet microscopy to provide simultaneous multiview imaging of abiological specimen;

FIG. 1B is a block diagram showing an exemplary schematic representationof the specimen imaged using the microscope system of FIG. 1A;

FIG. 2 is a block diagram of an exemplary implementation of themicroscope system of FIG. 1A;

FIG. 3 is a perspective view of an exemplary microscope of themicroscope system of FIG. 1A;

FIG. 4 is an expanded perspective view of a part of the exemplarymicroscope of FIG. 3 showing a specimen and objectives;

FIG. 5A is an expanded perspective view of an exemplary illuminationsystem of the microscope of FIG. 3;

FIG. 5B is a perspective view showing exemplary steps for preparing thespecimen and a chamber that holds the specimen in the microscope systemof FIG. 3;

FIG. 6 is a block diagram showing details of an exemplary electronicscontroller and an exemplary computational controller of the microscopesystem of FIG. 1A;

FIG. 7 is a flow chart of an exemplary procedure performed by themicroscope system of FIG. 1A;

FIGS. 8A-8D are block diagrams showing an exemplary imaging scheme inwhich the relative position of the specimen and the light sheets ismodified along a detection axis within the microscope system of FIG. 1A;

FIG. 9 is a flow chart of an exemplary procedure for creating an imageof the specimen as performed by the microscope system of FIG. 1A;

FIGS. 10A-10C are block diagrams of the microscope within the microscopesystem of FIG. 1A showing exemplary steps performed during the procedureof FIG. 9;

FIG. 11 is a diagram of a schematic representation of a specimen showingthe development of the specimen at exemplary points in time as the imageis captured using the microscope system of FIG. 1A;

FIGS. 12A and 12B are images of the specimen taken during a postprocessing step in the procedure of FIG. 7;

FIG. 13A are images of fluorescence taken with the microscope system ofFIG. 1A at steps along the detection axis using a one photon excitationscheme on a Drosophila embryo as the biological specimen;

FIG. 13B is a maximum-intensity projection of images of the Drosophilaembryo taken using the one photon excitation scheme at one point intime;

FIG. 13C are images of one region of the Drosophila embryo of FIG. 13Btaken in a time lapse series during the thirteenth mitotic cycle ofdevelopment;

FIG. 13D are maximum-intensity projections of the images of thefluorescence obtained during one-photon excitation using the microscopesystem of FIG. 1A on the Drosophila embryo of FIGS. 13A-13C;

FIG. 14A shows images of a Drosophila embryo taken at exemplary imagingplanes along the detection axis with the microscope system of FIG. 1Ausing a two-photon excitation scheme;

FIGS. 14B and 14C show maximum-intensity projections of a time-lapserecording of Drosophila embryonic development taken with the microscopesystem of FIG. 1A using the two-photon excitation scheme of FIG. 14A;

FIG. 15A shows raw image data superimposed with cell tracking results ofan entire Drosophila embryo taken before the twelfth mitotic wave andafter the thirteenth mitotic wave using the microscope system of FIG.1A;

FIG. 15B shows images of the cell-tracked Drosophila embryo of FIG. 15Ataken with the microscope system of FIG. 1A as the thirteenth mitoticwave progresses through the Drosophila embryo;

FIG. 15C shows an enlarged view of a reconstructed Drosophila embryo ofFIG. 15A with cell tracking information on the left and morphologicalnuclei segmentation on the right taken with the microscope system ofFIG. 1A;

FIG. 15D shows a graph of the average nucleus speed as a function oftime after nuclear division within the Drosophila embryo of FIGS.15A-15C;

FIG. 15E shows a graph of the distribution of the distances betweennearest nuclei neighbors within the Drosophila embryo of FIGS. 15A-15C;

FIG. 16A shows an image obtained using the microscope system of FIG. 1Ain a one-photon excitation scheme on a histone-labeled Drosophilaembryo, superimposed with manually reconstructed lineages of threeneuroblasts and one epidermoblast for 120-353 minutes postfertilization;

FIG. 16B shows an enlarged view of the tracks highlighted in FIG. 16A;

FIGS. 16C-E show maximum-intensity projections (dorsal and ventralhalves) of Drosophila embryonic nervous system development, recordedwith the microscope system of FIG. 1A using a one-photon excitationscheme;

FIG. 16F shows an enlarged view of the area highlighted in FIG. 16E;

FIG. 16G shows a progression of maximum-intensity projections of axonalmorphogenesis in a Drosophila transgenic embryo (false colorlook-up-table), recorded with the microscope system of FIG. 1A using aone-photon excitation scheme;

FIG. 17 is a block diagram of an exemplary microscope system in whicheach optical arm performs both illumination and detection tasks toprovide simultaneous multiview imaging of a biological specimen;

FIG. 18 is a block diagram showing details of an exemplary microscopesystem in which each optical arm performs both illumination anddetection tasks;

FIG. 19 is a perspective view of an optical microscope of the microscopesystem of FIG. 18;

FIG. 20 is a block diagram showing details of an electronics controllerand a computational system of the exemplary microscope system of FIG.18;

FIG. 21A is a block diagram showing an exemplary schematicrepresentation of the biological specimen imaged using one or more firstlight sheets produced in the microscope system of FIGS. 18-20;

FIG. 21B is a block diagram showing an exemplary schematicrepresentation of the biological specimen imaged using one or moresecond light sheets produced in the microscope system of FIGS. 18-20;

FIG. 22 is a flow chart of a procedure performed by the microscopesystem of FIGS. 18-20;

FIGS. 23A-C are block diagrams showing an exemplary imaging scheme inwhich the relative position of the specimen and the one or more firstlight sheets is modified along a z axis within the microscope system ofFIGS. 18-20;

FIG. 24 is a block diagram of the optical microscope within themicroscope system of FIGS. 18-20 showing a mode of operation during theprocedure of FIG. 22 that produces the imaging scheme shown in FIGS.23A-C;

FIGS. 25A-C are block diagrams showing an exemplary imaging scheme inwhich the relative position of the specimen and the one or more secondlight sheets is modified along a y axis within the microscope system ofFIGS. 18-20;

FIG. 26 is a block diagram of the optical microscope within themicroscope system of FIGS. 18-20 showing a mode of operation during theprocedure of FIG. 22 that produces the imaging scheme shown in FIGS.25A-C;

FIG. 27 is a block diagram of the optical microscope within themicroscope system of FIGS. 18-20 showing a multi-color mode of operationduring the procedure of FIG. 22;

FIG. 28 is a block diagram showing an exemplary imaging scheme in whichboth the one or more first light sheets and the one or more second lightsheets are scanned at the same time in the microscope system of FIGS.18-20;

FIG. 29 is a block diagram of one exemplary optical arm of themicroscope system of FIGS. 18-20;

FIG. 30 is a block diagram showing an exemplary imaging scheme in whichrecorded data in the microscope system of FIGS. 18-20 is separated usinga spatial separation;

FIGS. 31A-C are block diagrams showing an exemplary detection unit ofone of the optical arms of the microscope system of FIGS. 18-20 thatuses the imaging scheme of FIG. 30;

FIG. 32 is a block diagram showing how the light sheets of each opticalarm of the microscope system of FIGS. 18-20 align using the imagingscheme of FIG. 30;

FIG. 33 is a flow chart of an exemplary procedure performed by themicroscope system of FIGS. 18-20;

FIG. 34 is a block diagram of an exemplary microscope system based onthe microscope system of FIGS. 18-20 and including one or moreadditional optical arms that are non-parallel with the other opticalarms;

FIG. 35A is a block diagram of an exemplary microscope system that useslight sheet microscopy to provide simultaneous multiview imaging of abiological specimen and includes more than four optical views of thespecimen;

FIG. 35B is a perspective view of a microscope of the microscope systemof FIG. 17A showing illumination and detection with a first pair ofdetection subsystems;

FIG. 35C is a perspective view of the microscope of the microscopesystem of FIG. 17A showing illumination and detection with a second pairof detection subsystems after rotation of light sheets withinillumination subsystems; and

FIG. 36 is a flow chart of an exemplary procedure performed by themicroscope system of FIG. 17A to image the biological specimen usingmore than four optical views.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

DETAILED DESCRIPTION

Referring to FIG. 1A, this description relates to a microscope system100 and corresponding process for live imaging of a complex biologicalspecimen (or specimen) 101, such as a developing embryo, in itsentirety. For example, the complex biological specimen 101 can start offas a fertilized egg; in this case, the microscope system 100 can capturethe transformation of the entire fertilized egg into a functioninganimal, including the ability to track each cell in the embryo thatforms from the fertilized egg as it takes shape over a period of time onthe scale of hours or days. The microscope system 100 can provide acompilation of many images captured over about 20 hours to enable theviewer to see the biological structures within the embryo that begin toemerge as a simple cluster of cells morph into an elongated body withtens of thousands of densely packed cells.

The microscope system 100 uses light-sheet microscopy technology thatprovides simultaneous multiview imaging, which eliminates or reducesspatiotemporal artifacts that can be caused by slower sequentialmultiview imaging. Additionally, because only a thin section (forexample, on the order of a micrometer (μm) wide taken along the z axis)of the specimen 101 is illuminated at a time with a scanned sheet oflaser light while a detector records the part of the specimen 101 thatis being illuminated, damage to the specimen 101 is reduced. Nomechanical rotation of the specimen 101 is required to perform thesimultaneous multiview imaging.

In general, the optical microscope 110 is made up of a plurality oflight sheets (for example, light sheets 102, 104) that illuminate thespecimen 101 from distinct directions along respective light sheet axes,and a plurality of detection subsystems (for example, detectionsubsystems 116, 118) that collect the resulting fluorescence along aplurality of detection views. In the example that follows, two lightsheets 102, 104 are produced in respective illumination subsystems 112,114, which illuminate the specimen 101 from opposite directions or lightsheet axes; and the respective detection subsystems 116, 118 collect theresulting fluorescence along two detection views. In this particularexample, the light sheet axes are parallel with an illumination axis(the y axis) and the detection views are parallel with a detection axis(the z axis), which is perpendicular to the y axis.

Therefore, in this example, the microscope system 100 providesnear-complete coverage with the acquisition of four complementaryoptical views; the first view comes from the detection system 116detecting the fluorescence emitted due to the interaction of the lightsheet 102 with the specimen 101; the second view comes from thedetection system 116 detecting the fluorescence emitted due to theinteraction of the light sheet 104 with the specimen 101; the third viewcomes from the detection system 118 detecting the fluorescence emitteddue to the interaction of the light sheet 102 with the specimen 101; andthe fourth view comes from the detection system 118 detecting thefluorescence emitted due to the interaction of the light sheet 104 withthe specimen 101.

Referring to FIG. 1B, which is exaggerated to more clearly show theinteractions between the light sheets 102, 104, and the specimen 101,the light sheets 102, 104 spatially overlap and temporally overlap eachother within the specimen 101 along an image volume IV that extendsalong the y-x plane, and optically interact with the specimen 101 withinthe image volume IV. The temporal overlap is within a time shift ordifference that is less than a resolution time that corresponds to thespatial resolution limit of the microscope 110. In particular, thismeans the light sheets 102, 104 overlap spatially within the imagevolume IV of the biological specimen 101 at the same time or staggeredin time by the time difference that is so small that any displacement oftracked cells C within the biological specimen 101 during the timedifference is significantly less than (for example, an order ofmagnitude below) a resolution limit of the microscope 110, where theresolution limit is a time that corresponds to a spatial resolutionlimit of the microscope 110.

As will be discussed in greater detail below, each light sheet 102, 104is generated with a laser scanner that rapidly moves a thin (forexample, μm-thick) beam of laser light along an illumination axis (the xaxis), which is perpendicular to the y and z axes, to form a light beamthat extends generally along or parallel with a plane to form the sheet102, 104. In this example, the laser beam in the form of the light sheet102, 104 illuminates the specimen 101 along the y axis on opposite sidesof the specimen 101. Rapid scanning of a thin volume and fluorescencedetection at a right angle (in this example, along the z axis) to theillumination axis provides an optically sectioned image. The lightsheets 102, 104 excite fluorophores within the specimen 101 into higherenergy levels, which then results in the subsequent emission of afluorescence photon P, and the fluorescence photons P are detected bythe detectors within the detection subsystems 116, 118 (along the zaxis). As discussed in detail below, in some implementations, theexcitation is one-photon excitation, or it is multi-photon (for example,two-photon) excitation.

The fluorophores that are excited in the specimen can be labels that areattached to the cells, such as, for example, genetically-encodedfluorescent proteins such as GFP or dyes such as Alexa-488. However, thefluorophores can, in some implementations that use second-harmonicgeneration or third-harmonic generation, be actual or native proteinswithin the cells that emit light of specific wavelengths upon exposurewith the light sheets 102, 104.

As shown schematically in FIG. 1B, the light sheets 102, 104 passthrough the specimen 101 and excite the fluorophores. However, the lightsheets 102, 104 are subject to light scattering and light absorptionalong their respective paths through the specimen 101. Moreover, verylarge (large compared with the image volume IV or the field-of-view(FOV)) or fairly opaque specimens can absorb energy from the lightsheets 102, 104.

Moreover, if the light sheets 102, 104 are implemented in a two-photonexcitation scheme, then only the central region 103 of the overlappinglight sheets 102, 104 may have a high enough power density toefficiently trigger the two-photon process, and it is possible that only(the close) half of the specimen 101 emits fluorescence photons P inresponse to exposure to two-photon light sheets 102, 104.

The term “spatial overlap” of the light sheets could mean that the lightsheets 102, 104 are overlaid geometrically within the specimen 101. Theterm “spatial overlap” can also encompass having the light sheets botharrive geometrically within the FOV (as shown in FIG. 1B) of thedetection subsystems 116, 118 and within the specimen 101. For example,to efficiently trigger two-photon excitation, each light sheet 102, 104could cover only a part of (for example, one half) of the field-of-viewof the detection subsystems 116, 118 (so that each light sheet iscentered in the respective half of the field-of-view) so that the use ofboth of the light sheets 102, 104 leads to the full field-of-view beingvisible.

Because of this, each two-photon light sheet 102, 104 can be madethinner (as measured along the z axis), if the light sheet 102, 104 onlyneeds to cover half of the field-of-view. However, if the light sheet102, 104 is thinner (and the laser power unchanged), the same number ofphotons travel through a smaller cross-section of the specimen 101, thatis, the laser power density is higher, which leads to more efficienttwo-photon excitation (which is proportional to the square of the laserpower density). At the same time, because the light sheets are thinner,the resolution is increased when compared to a scenario in which eachlight sheet 102, 104 covers the entire field-of-view.

As another example, if the light sheets 102, 104 are implemented in aone-photon excitation scheme, the light sheet 102, 104 could also excitefluorophores on the area of the specimen 101 outside of the centralregion 103 but within the image volume, and this part will appearblurrier in the resultant image. In this latter case, two images can besequentially recorded with each of the two light sheets 102, 104, andthe computational system 190 can use a calculation to adjust the imagesto obtain a higher quality. For example, the two images can be croppedsuch that the low-contrast regions are eliminated (and complementaryimage parts remain after this step), and then the images recorded withthe two light sheets can be stitched together to obtain a final imagethat covers the entire field-of-view in high quality.

The light sheet 102, 104 is configured so that its minimal thickness orwidth (as taken along the z axis) is within the image volume IV and theFOV. When two light sheets 102, 104 are directed toward the specimen101, then the minimal thickness of the respective light sheet 102, 104should overlap with the image volume IV. As discussed above, it can beset up so that the minimal thickness of the light sheet 102 is offsetfrom the minimal thickness of the light sheet 104, as shownschematically in FIG. 1B. This set up provides improved or superiorspatial resolution (for both one-photon and two-photon excitationschemes) and improved or superior signal rates (for two-photonexcitation schemes).

For example, for a specimen 101 that is a Drosophila embryo that isabout 200 μm thick (taken along the z axis), the light sheet 102 can beconfigured to reach its minimal thickness about 50 μm from the left edge(measured as the left side of the page) of the specimen 101 after itcrosses into the specimen 101 while the light sheet 104 can beconfigured to reach its minimal thickness about 50 μm from the rightedge (measured as the right side on the page) of the specimen 101 afterit crosses into the specimen 101.

There is a tradeoff between the minimal thickness of the light sheet102, 104 and the uniformity of the light sheet 102, 104 thickness acrossthe image volume IV. Thus, if the minimal thickness is reduced, then thelight sheet 102, 104 becomes thicker at the edges of the image volumeIV. The thickness of the light sheet 102, 104 is proportional to thenumerical aperture of the respective illumination subsystem 112, 114;and the useable length of the light sheet 102, 104, that is, the lengthover which the thickness is sufficiently uniform, is inverselyproportional to the square of the numerical aperture. The thickness ofthe light sheet 102, 104 can be estimated using any suitable metric,such as the full width of the light sheet 102, 104 along the z axistaken at half its maximum intensity (FWHM).

For example, a light sheet 102, 104 having a minimal thickness of 4 μm(using a suitable metric such as the FWHM) is a good match for an imagevolume IV that has a FOV of 250 μm (which means that it is 250 μm long(as taken along the y axis)). A good match means that it provides a goodaverage resolution across the field of view. A thinner light sheet wouldimprove resolution in the center (taken along the y axis) but coulddegrade the resolution dramatically and unacceptably at the edges of thespecimen 101, and possibly lead to worse average resolution across thefield-of-view. A thicker light sheet would make the light sheet moreuniform across the image volume IV, but it would degrade the resolutionacross the entire image volume IV. As another example, a light sheet102, 104 having a minimal thickness of 7 μm (using a suitable metricsuch as the FWHM) is a good match for an image volume IV that has a FOVof 700 μm (and thus it is 700 μm long (as taken along the y axis)).

In general, the thickness of the light sheet 102, 104 (as taken alongthe z axis) should be less than the size (and the thickness) of thespecimen 101 to maintain image contrast and reduce out of focusbackground light. In particular, the thickness of the light sheet 102,104 should be substantially less than the size of the specimen 101 inorder to improve the image contrast over a conventional illuminationapproach in which the entire specimen 101 is illuminated. Only in thisregime (light sheet thickness is substantially smaller than specimenthickness), light-sheet microscopy provides a substantial advantage overconventional illumination approaches.

For example, the thickness of the light sheet 102, 104 can be less thanone tenth of the width of the specimen 101 as taken along the z axis. Insome implementations, the thickness of the light sheet 102, 104 is onthe order of one hundredth of the width or size of the specimen 101 ifeach light sheet is used to cover only a portion (such as a half) of thefield-of-view, that is, the point of minimal thickness of each lightsheet is located in the center of one of the two halves of thefield-of-view, as discussed above.

Another important consideration for setting the minimal thickness of thelight sheet 102, 104 or for defining a realistic image volume IV, is thesize of the structure within the specimen 101 that needs to be resolvedby the microscope system 100. Often, the microscope system 100 is set upto image cell nuclei, which are about 4-5 μm in size (taken along astraight line across the nucleus) in Drosophila, and around 7-8 μm insize in a zebrafish at the early developmental stages in which thesystem 100 can be used. To achieve reasonably good spatial sampling andresolution, the light sheet 102, 104 should have a minimal thicknessthat is not much thicker than half the cross-sectional size or length ofthese nuclei. Thus, for Drosophila, the minimal thickness of the lightsheet 102, 104 should be less than 2-3 μm while for the zebrafish, theminimal thickness of the light sheet 102, 104 should be less than 3-4μm. Moreover, the images of the specimen 101 should be recorded in stepstaken along the z axis (by either translating the specimen 101, thelight beams 102, 104, or both the specimen 101 and the light beams 102,104 along the z axis); and the size of the steps should be about thesize of this minimal thickness.

The microscope system 100 is a comprehensive solution that includes acomplete technology framework. Thus, the microscope system 100 alsoincludes an electronics framework that includes the electronicscontroller 180 and the computational system 190. The electronicscontroller 180 provides synchronized control of all opto-mechanicalcomponents within the microscope 110 with millisecond precision overlong periods of time. The computational system 190 rapidly performs thecomplex optical alignment on the live specimen 101, and provides arobust pipeline for simultaneous high-speed image acquisition with aplurality of detectors (or cameras) within the detection subsystems 116,118 at sustained data rates of several hundreds of megabytes per secondand a high-throughput computational strategy for efficient automatedimage processing to register and reconstruct the terabytes of rawmultiview image data arising from every experiment.

While the example provided herein describes a four-view system using twolight sheets for illumination and two detectors for collecting thefluorescence, more views are possible by adding additional detectorsand/or illumination systems. In this example, the illuminationsubsystems 112, 114 face each other such that the specimen 101 can beilluminated with light sheets from two sides, as more clearly shown inthe schematic drawing of FIG. 1B. Each detection subsystem 116, 118includes a respective detector or camera 106, 108 in addition to a setof detection optical devices arranged to collect and record thefluorescence emitted from the specimen 101. Each detection subsystem116, 118 also includes a set of actuators that are coupled to one ormore of the detector 106, 108 and the detection optical devices at oneend and interface at the other end with the electronics controller 180.

Each combination of illumination and detection provides a different viewor perspective. By capturing the four views (in this example)simultaneously, delays caused by rotating of the specimen 101 repeatedlyinto new positions (as in the sequential imaging techniques of past) arereduced or eliminated. Thus, the microscope system 100 is designed forthe simultaneous acquisition of multiple complementary views withouthaving the rotate the specimen 101.

Illumination Subsystems

Referring also to FIGS. 2-5A, each of the illumination subsystems 112,114 includes respective light sources 122, 124 that output respectivelight beams, and respective sets 132, 134 of optical components thatmodify properties such as direction, size, geometry, etc. of the lightbeam to produce the light sheets 102, 104 that are directed to thespecimen 101. The light sheets 102, 104 produced within the illuminationsubsystems 112, 114 can be produced by scanning an output of therespective light sources 122, 124, as will be discussed below.

The light sources 122, 124 can include one or more light sources such aslasers. In some implementations, the light sheets 102, 104 can begenerated from a single (one) light source, and then the output fromthis single light source can be split into two beams to operate as therespective light sources 122, 124. While in other implementations, thelight sheets can be generated from two separate light sources thatoperate as the respective light sources 122, 124.

For a two-photon excitation arrangement, a single pulsed Ti:Sapphirelaser can be the sole light source that is split into two beams. In thiscase, the wavelength of the light can be adjusted to any value between690 and 1080 nm.

In some implementations of a one-photon excitation arrangement, thelight source can be made up of six laser diodes and diode pumped solidstate (DPSS) lasers, whose laser beams are all combined on the sameoptical axis using dichroic mirrors, thus effectively turning the entireunit into something that appears to be a single light source. In thiscase, there is a benefit to using more than one light source because itprovides for multi-color imaging or increased flexibility in fluorescentmarker selection. The laser diodes and the DPSS laser each provide adistinct wavelength, and this enables the microscope system 100 to workwith different types of fluorescent proteins and even multiple colorchannels within a single experiment. In some implementations, at anygiven time, only one wavelength of the arrangement may be activated. Theactivation of the wavelength can be controlled by the electronicscontroller 180 and the computational system 190; for example, theelectronics controller 180 can send a signal to the lasers oracousto-optical tunable filters, which activate or de-activate thecontribution of the respective wavelength within microsecond speed.Since the laser beam is then split into two beams using a static opticalelement such as a beam splitter, both of the illumination subsystems112, 114 receive light at the same time; and the fast laser shutters ineach subsystem 112, 114 enable selective activation of the light sheets102, 104 produced in the respective subsystem 112, 114.

In other implementations, a dedicated laser system (or light source) canbe provided for each illumination subsystem, and each dedicated lasersystem could be individually controllable by the electronics controller180 and the computational system 190.

For example, the light source 122 and the light source 124 can eachinclude one or more solid-state lasers (such as diode-pumped solid-statelasers) that operate at a plurality of wavelengths to provide forflexibility. In one example, each light source 122, 124 includes a SOLE®Laser Light Engine produced by Omicron-Laserage Laserprodukte GmbH ofRodgau-Dudenhofen, Germany. Details on such lights sources can be foundat their website at www.omicron-laser.de/. In these Light Engines, aplurality of outputs from internal and individual lasers operating atdistinct wavelengths can be combined to form a single output. Outputsfrom the SOLE® Laser Light Engines can be directed through one or morefiber optics, depending on the application. The light sources 122, 124are connected to components within the electronics controller 180 toenable the control of the active laser lines and their respective laserpower. Additionally, after being combined to form a single output, it ispossible to split this output into two illumination arms forfluorescence excitation from near-UV to near-IR, providing the laserwavelengths at, for example, 405 nm, 445 nm, 488 nm, 515 nm, 561 nm, 594nm, 642 nm, and 685 nm.

In other implementations, it is possible to use a plurality of lightsources and/or a plurality of light sheets having different colors atany given time (simultaneously), and equip each detection subsystem 116,118 with dichroic beam splitters and a plurality of cameras, such thatdifferent color channels can be recorded simultaneously.

The output from each of the light sources 122, 124 can be controlledusing optical shutters 123, 125, which control the timing of the lightsheets 102, 104 that reach the specimen 101, and thus control which ofthe light sheets 102, 104 (for example, only the light sheet 102, onlythe light sheet 104, or both the light sheet 102 and light sheet 104)illuminate the specimen 101 at any one moment. In some implementations,the optical shutters 123, 125 are Uniblitz laser shutters (such asUniblitz LS6ZM2-100 laser shutters or Uniblitz VS14S2ZM1-100 lasershutters). These optical shutters are able to block light that isdirected through an aperture, and the blocking of the light can besynchronized, asychronized, or controlled depending on the application.The illumination at the specimen 101 can therefore be performedsynchronously or asynchronously from either side. Respective drivers oractuators 126, 128 operate the laser shutters 123, 125 under controlfrom the electronics controller 180. The actuators 126, 128 can be, forexample, Uniblitz VMM-D3 three-channel drivers.

Other types of light sources are possible, and the constraints on theselection of the light source include desired wavelength or wavelengths(which can be selected or modified to excite the fluorophores within thespecimen 101), desired power, and desired beam quality.

The light beams that are output from the light sources 122, 124 aredirected through the respective sets 132, 134 of optical components,which are driven by respective actuator systems 133, 135 that areconnected to the electronics controller 180. In general, each set 132,134 of optical components can include, for example, mirrors, lenses, andobjectives, opto-mechanical components (such as objectives) andelectro-optical components.

For clarity, the optical component set 132 and its actuator system 133are described next, and the other optical component set 134 and itsactuator system 135 are similarly designed so a separate description ofthose elements are not provided.

The laser light exits the light source 122 in the illumination subsystem112 through the optical shutter 123, and is directed to an opticalscanner device 140, which deflects the light beam along the x axis toform a light sheet, which is directed toward an f-theta lens 141, a tubelens 142, and an illumination objective 143. The optical scanner device140 deflects the incident laser light to produce an angular range thatdefines the height of the light sheet 102 along the x axis in thespecimen 101. And, the angle of the light beam that exits the opticalscanner device 140 is converted into a displacement along the x axis ordirection by the f-theta lens 141 because the optical scanner device 140is positioned to be located at the focal plane of the f-theta lens 141,thus producing a macroscopic light sheet. The pair of the tube lens 142and the illumination objective 143 focuses the macroscopic light sheetoutput from the f-theta lens 141 into the microscopic light sheet 102 atthe specimen 101. The illumination objective 143 can be a relativelylong working distance air objective. The illumination objective 143 canbe a microscope objective that includes a single lens, or a combinationsof lenses and other optical elements.

In this description, the optical scanner device 140 deflects the lightbeam along the x axis; however, it is possible that the light beam canbe deflected along another axis, depending on the application. Forexample, when performing volumetric imaging without moving the specimen101, the light beam can be deflected along the z axis too. Moreover, itis possible to deflect or more the light beam along the z axis to alignthe light sheets 102, 104 with respect to each other.

The optical scanner device 140 can be formed by one or more moveablemirrors 140. The mirrors 140 can be mounted on a tip/tilt stage that ismoveable under control of an actuator such as a piezoelectric driver, orthey can be integrated within an actuator system such as the S-334miniature piezo tip/tilt mirror by Physik Instrumente (PI), whichincludes the mirror. As another implementation, the mirrors 140 can becontrolled using galvanometer scanners, such as the model 6215HSM40Bfrom Cambridge Technology. Combining two scanners in an XY scan headprovides the same angular degrees of freedom as a tip/tilt mirrorassembly, but can be faster.

The piezoelectric driver can be connected to a piezo driver module and apiezo servo module, which are within the electronics controller 180. Forexample, the PI S-334 can be connected to a PI E-503 piezo amplifiermodule that includes amplifiers that output current to the piezoelectricactuator to control the movement of the mirror 140 and to a PI E-509piezo servo module. All of the modules for controlling the mirrors 140can be housed within a sub-control module within the electronicscontroller 180.

The f-theta lens 141 converts the tilting movement of the scan mirror140 into a displacement (shown by the arrow in FIG. 5A) of the laserbeam along the x axis. The tube lens 142 and the illumination objective143 focus the laser beam into the specimen 101, which is aligned withthe detection subsystems 116, 118 along the z axis in this example. Thef-theta lens 141 can be, for example, a color-corrected lens such aspart S4LFT4375 from Sill Optics, which operates for light having awavelength between 450 nm and 650 nm. The tube lens 142 and theillumination objective 143 can be purchased off the shelf from supplierssuch as Carl Zeiss, Nikon, or Olympus. In one example, the tube lens 142is an Olympus U-TLU-1-2 camera tube lens and the illumination objective143 is an Olympus XLFLUOR 4x/340/0.28 objective mounted on apiezoelectric positioner that is controlled by electronics within theelectronics controller 180. For example, the illumination objective 143can be mounted to a long-travel scanner such as the P-725 PIFOC producedby Physik Instrumente, which is controlled by an amplifier/servocontroller, such as the E-665 produced by Physik Instrumente, within theelectronics controller 180.

Additionally, while not shown, the light beams that are output from thelight sources 122, 124 can be directed through respective opticalfilters, which can be mounted on a compact illumination filter wheelwith controller (such as a DC servo controller made by Ludl ElectronicsProducts Ltd. of Hawthorne, N.Y.).

In some implementations, the electronic components required to operateall elements in the illumination subsystem 112 can be controlled by afirst set of components within the electronics controller 180, while theelectronics components required to operate all the elements in theillumination subsystem 114 can be controlled by a second set ofcomponents within the electronics controller 180. The selection of thespecific optical components in the sets 132, 134 depend on thewavelength of the light that must be passed and directed, the power atwhich the light operates, and other factors such as numerical apertureor size of focus.

Detection Subsystems

Next, exemplary detection subsystems 116, 118 will be discussed ingreater detail. In some implementations, as shown in FIG. 3, both of thesubsystems 116, 118 are designed with the same components. For example,the fluorescence light (or photons P) emitted from the specimen 101 iscollected by opposing detection objectives 150, 152, and the collectedlight from respective objectives 150, 152 passes through a filter 154,156, which rejects light of wavelengths outside a wavelength bandcentered around the wavelength of the fluorescence light to be detected.The filter 154, 156 can be mounted on a filter wheel (not shown) withother filters, so that the wavelength of rejected light could be changedby selecting a different filter on the wheel by merely rotating thewheel into a new position. The filters 154, 156 can be short-pass orband-pass filters, such as, for example, BrightLine fluorescence filtersfrom Semrock, Inc, a part of IDEX Corporation of Lake Forest, Ill.

Light from the filter 154, 156 is directed through respective lenses158, 160, which focus the light onto the sensor of the camera 106, 108.The lenses 158, 160 can be tube lenses.

The detection objectives 150, 152 can be microscope objectives that eachinclude a single lens, or a combinations of lenses and other opticalelements. In some implementations, the detection objectives 150, 152 canbe high numerical aperture water-dipping objectives. For example, theobjectives can be purchased from suppliers such as Carl Zeiss, Nikon, orOlympus. The detection objectives 150, 152 can be mounted or attached toscanners 162, 164, which are connected to and controlled by theconnector blocks within the electronics controller 180. For example, thescanner can be a piezo-actuated scanner (a nanopositioner) such as thePIFOC® PI P-725 PIFOC long-travel objective scanner produced by PhysikInstrumente (PI) GmbH & Co. KG of Germany.

The filter wheel can be purchased from Ludl Electronic Products Ltd. Forexample, the filter wheel can be the 96A354 6-slot filter wheel producedby Ludl. The filter wheels are operated by one or more servocontrollers, which are connected to connector blocks within theelectronics controller 180.

The tube lenses 158, 160 can be purchased from suppliers such as CarlZeiss, Nikon, or Olympus, and the cameras 106, 108 can be scientificCMOS (complementary metal oxide semiconductor) image sensors (sCMOSsensors). The sCMOS sensors can be purchased from Andor Technology plc.or from PCO-TECH Inc. (formerly The Cooke Corporation). Or, the sCMOSsensors can be the Orca Flash 4.0 sCMOS sensor from Hamamatsu. Thecameras 106, 108 are connected to full-configuration frame grabbers inthe computational system 190 by a pair of standard data transfer cables,such as Camera Link® cables, that are a part of the electronicscontroller 180.

Specimen and Supporting Components

Next, a description of the specimen 101 and its associated optical,mechanical and electrical components is provided.

Referring also to FIGS. 5A and 5B, the specimen 101 is attached to aspecimen holder 166 that enables the specimen 101 to be opticallyaccessible to all of the illumination subsystems 112, 114 and thedetection subsystems 116, 118. The specimen 101 and the holder 166 canbe placed within a hollow space defined by a specimen chamber 168. Forexample, the holder 166 can include a glass capillary 200 that ismounted to a solid base 201. The base 201 of the specimen holder 166 canbe produced from medical-grade stainless steel. The holder 166 can be acustom water-sealed flexible specimen holder.

The holder 166 is physically attached to a positioning system thatprovides a translation stage and a rotation stage, which is connected tothe electronics controller 180. The translation stage can be set up totranslate the holder 166 and thus the specimen 101 along threeorthogonal directions, for example, along the x, y, and z axes. Therotation stage can be set up to rotate the holder 166 and thus thespecimen about the x axes. The positioning system can be inside of orexternal to the specimen chamber 168.

One or more of the holder 166 and the chamber 168 can be connected to athermoelectric cooling device such as an integrated Peltier cooler,heater, or thermoelectric heat pump to maintain the specimen 101 at aparticular temperature.

The objectives in the illumination subsystems 112, 114 and the detectionsubsystems 116, 118 are directed at the center region of the chamber168, at which the specimen 101 is positioned and located. In thisimplementation, the axis (which is the y axis) of the illuminationsubsystems 112, 114 is oriented perpendicularly to the axis (which isthe z axis) of the detection subsystems 116, 118.

In some implementations, the positioning system for the holder 166 (andthus the specimen 101) can be a high performance stepper and servomotion control such as the NI PXI-7354 motion controller produced byNational Instruments. This control can be connected to an interfacewithin the electronics controller 180 such as a C-809.40 4-channelservo-amplifier/motion I/O interface produced by Physik Instrumente (PI)GmbH & Co. KG.

The chamber 168 has an internal hollow space having a volume that islarge enough to accommodate the specimen 101 and the holder 166; forexample, a chamber with full optical access from all sides to enablesimultaneous light sheet illumination and fluorescence detection withfour objectives. Full optical access and improved specimen positioningcontrol (three-dimensional translation and one-dimensional rotation) canbe realized by splitting the microscope sub-systems into two horizontallayers. For example, all optical systems could be located in an upperlayer, whereas the specimen positioning system could be locatedunderneath the specimen chamber 168 on a lower optical table. Thespecimen positioning system can be configured in an upright position(for example, along the x axis), thus permitting the use ofexceptionally soft (0.4%) agarose gels for specimen embedding within theglass capillary 200 of the holder 166. Mechanical stability duringimaging is largely provided by the solid base of the holder 166 on whichthe capillary is mounted.

The chamber 168 can be custom designed and built to provide enough viewsfor the illumination and detection; in this example, four views. It caninclude a perfusion system. The specimen chamber 168 houses the specimenholder 166, and can also house a multi-stage adapter module forconnecting the specimen holder 166 to the specimen positioning systemand the custom perfusion system. Using the positioning system, theholder 166 can be translated in three dimensions and rotated around itsmain axis (the x axis) without breaking the water seal. The chamber 168is open to two opposite sides to accommodate the water-dipping detectionobjectives and contains two windows with coverslips on the remaining twosides for laser light sheet illumination. The top of the chamber 168 isopen for mechanical or optical access and allows background illuminationwith a cold light source. The chamber 168 has inlet and outlet valvesconnected to the perfusion system, which is operated by a dual-channel12-roller pump (for example, a REGLO by Harvard Apparatus of Holliston,Mass.). The pump can be connected to the chamber 168 by a clear,flexible tubing having some level of resistance to chemicals, such asTygon™ tubing, and the tubing is guided through a bench-top incubator(Model 107 bench-top environmental chamber, TestEquity) for temperaturecontrol and oxygenation of the specimen 101.

In some implementations, the specimen positioning system can bemagnetically connected to the specimen holder 166 in the chamber 168.

As mentioned above, the chamber 168 can also include a custom perfusionsystem, which is a pump that enables the replacement of the buffer vaporin the cavity of the chamber 168 at a desired rate, to maintainphysiological conditions for the specimen 101. Such a perfusion systemcan be beneficial for long-term imaging experiments with specimens 101that can be difficult to culture, such as, for example, mouse embryos,or for imaging that requires stable environmental conditions, such astemperature, pH, oxygen concentration, sugars and amino acids insurrounding medium. The perfusion system can be used in combination witha temperature control system that is thermally connected to the chamber168 to permit temperature regulation of the chamber and the environmentaround the specimen 101.

Electronics Controller

Referring to FIG. 6, all optical and mechanical components (such aswithin the illumination subsystems 112, 114, the detection subsystems116, 118, or the specimen holder 166 and specimen chamber 168) of themicroscope 110 are operated and synchronized by a high-performancereal-time electronics controller 650 provided within the electronicscontroller 180. The real-time controller 650 communicates with thecomputational system 190 to coordinate the simultaneous imageacquisition workflow at a rate of 350 megabyte/second. The electronicscontroller 180 can also include special camera links 660 for connectingdirectly to the cameras 106, 108.

All of the optical, mechanical, and electrical components within theoptical microscope 110 are connected to components within the controller650 to provide real-time control in each of the “arms” of themicroscope; with each arm being one of either the illuminationsubsystems 112, 114 or the detection subsystems 116, 118. The controller650 also includes an automated alignment module for rapid relativepositioning and orientation of the two light sheets 102, 104 and therespective focal planes. The controller 650 modifies the variousoptical, mechanical, and electrical components of the optical microscopeto provide twenty degrees of freedom of motion to optimize thefluorescence signal.

In one general aspect, the real-time electronics controller 650 is areal-time controller such as a PXI-8110 2.2 GHz Quad Core embeddedcontroller by National Instruments Corporation of Austin, Tex. Thiscontroller can run a LabVIEW Real-Time operating system, and is equippedwith three I/O interface boards (such as the PXI-6733 high-speed analogoutput 8-channel board, also by National Instruments) linked to BNCconnector blocks (such as the BNC-2110 shielded connector block, also byNational Instruments) as well as a serial interface board (such asPXI-8432/2, also by National Instruments). The real-time controller 650communicates with the computational system 190 by way of a high-speeddata transmission such as Gigabit Ethernet.

In addition to the real-time controller 650, the electronics controller180 can also include a motion controller and analog and digitalinput/output channels on separate controllers. The motion controller canbe a PXI-7354 motion controller from National Instruments and it caninclude a plurality of I/O controllers such as the PXI-6733 controllersfrom National Instruments, having eight analog outputs and eight digitalI/O channels each.

All time-critical tasks can be performed within the electronicscontroller 180, with the remaining tasks (such as collecting andvisualizing frames recorded by the cameras 106, 108) being assigned tothe computational system 190.

Computational System

Referring again to FIG. 6, the computational system 190 can include acomputer such as a workstation that has the ability to store, retrieve,and process data. Thus, the computer includes hardware such as one ormore output devices 600 such as a monitor or a printer; one or more userinput interfaces 602 such as a keyboard, a mouse, a touch display, or amicrophone; one or more processing units 604, including specializedworkstations for performing specific tasks; memory (such as, forexample, random-access memory or read-only memory or virtual memory)606; and one or more storage devices 608 such as hard disk drives, solidstate drives, or optical disks. The processing units can be stand aloneprocessors, or can be sub-computers such as workstations in their ownright.

The specialized workstations include an on board processing unit, inaddition to memory, and software for running specific tasks. Thespecialized workstations includes an image acquisition workstation 610,an image processing workstation 620, and an optional image segmentationand tracking workstation 630.

The storage devices 608 include, among others, an image data managementstorage unit 618 that receives information from the image acquisitionworkstation 610 and also is equipped to receive its own processors, foradditional processing capabilities.

The workstations include their own software modules, which include a setof instructions that tells the hardware within the electronicscontroller 180 what to do.

The computational system 190 is designed for high-speed imagingexperiments with up to several days of continuous image acquisition. Thecomputational system 190 is capable of recording more than one millionhigh-resolution images in uninterrupted high-speed imaging sessions witha total data set size of up to ten terabytes per specimen. In order topermit high-speed recording, each of the lines connecting the imageacquisition workstation 610 to the image data management unit 618 and tothe controller 650 can be set up as a glass fiber network pipeline,providing 10 gigabit/second data speed thus allowing recording up to tenmillion high-resolution images or 100 terabytes per specimen 101 forlong-term imaging sessions. A maximum recording capacity of one petabytecan be realized if the image data management unit 618 uses athree-dimensional wavelet compression technique having an average ratioof 10:1.

The image acquisition workstation 610 and the multiview image processingworkstation 620 are developed for content-based image registration andmultiview image fusion, respectively, which efficiently incorporatesprior knowledge of the optical implementation to process raw image dataat a rate of about 200 megabyte/seconds. The image acquisitionworkstation 610 is capable of real-time image registration andintegrates with the image processing workstation 620 for large-scaledata management. Since the computational system 190 acquires multipleviews simultaneously, fast and accurate image registration (alignment ofimages) within the image acquisition workstation 610 is achieved withoutthe need of fiducial markers in the imaging volume IV.

In one implementation, the computational system 190 is a Windows-basedpersonal computer having twelve physical core processors within theprocessing units 604 operating at 3.3 GHz and accessing 64 gigabytes(GB) of RAM in memory 606. The processing unit 604 can run a LabVIEW®Developer Suite. The memory 606 can also contain a fast RAID system ofhard disk drives having a total capacity of 2×5 TB (22 hard disks with600 GB capacity each, combined in two RAID-6 arrays with two diskredundancy each) and 2 NI PCIe-1429 full-configuration frame grabbers.Having two virtual disk drives allows assigning one drive to each camera106, 108 or writing images in an alternating manner on both drives usingonly a single camera 106, 108. Typically, however, both cameras 106, 108are used in an experiment.

In the following, we provide a short description of the workflow in thecomputational system 190, referring to the z axis as the optical axis ofthe detection subsystems 116, 118, the y axis as the optical axis of theillumination subsystems 112, 114, and the x axis as the remaining axisto form a Cartesian coordinate system.

In general, the image acquisition workstation 610 performs aregistration of images, which is a process that aligns the images fromthe cameras 106, 108. In general, the image processing workstation 620performs a fusion of the images, which is a process of combining theregistered images into a single representation or image. In particular,images are fused by combining the information content of the images intoa single image. Details about fusion are discussed below.

The image data management unit 618 provides a high-throughput imagestorage pipeline for sustained data streaming at 600 megabytes/second;uninterrupted long-term image acquisition of 100 terabyte sized datasets, and a wavelet-based lossless image compression (for example,ten-fold).

In some implementations, the image acquisition workstation 610 includesa pair of 6-core central processing units (CPUs), one for each of thecameras 106, 108, for processing the images. The CPUs can be, forexample, Xeon® Processors X5680 purchased from Intel Corporation ofSanta Clara, Calif. The image acquisition workstation 610 can alsoinclude dedicated memory such as RAM (for example, 144 GB DDR-3 RAM fromKingston) for image ring buffers and online processing, a 24-channelRAID controller (for example, model 52445 from Adaptec) with 22 SAS-2hard disks (for example, the Cheetah 15K.7 by Seagate) combined in aRAID-6 for high-speed image acquisition, a 10 Gigabit fiber networkadapter (such as, for example, EXPX9501AFXSR, Intel Corporation) foronline data streaming, a graphics adapter (such as, for example, GeForceGTX470, Nvidia Corporation) for GPU-based processing, two Camera Linkframe grabbers (such as, for example, Neon, BitFlow; or PCIe-1429,National Instruments) and a server board (such as, for example,X8DAH+-F, Supermicro).

The image acquisition workstation 610 relays the raw multiview datastream to the image processing workstation 620 and the image datamanagement unit 618 by way of optical fibers. The software used in theworkstation 610 can be written in, for example, Matlab and C++ forproviding high-throughput multiview image processing and real-time imagedata management.

In some implementations, the image processing workstation 620 includes apair of 6-core CPUs (such as, for example, Xeon® Processors X5680purchased from Intel Corporation of Santa Clara, Calif.), 96 GB DDR-3RAM (Kingston), a 16-channel RAID controller (51645, Adaptec) with 2SAS-2 hard disks (AL11SE 147 GB, Toshiba) combined in a RAID-1 for theoperating system and 10 SAS-2 hard disks (AL11SE 600 GB, Toshiba)combined in a RAID-6 for image data buffering, a 10 Gigabit fibernetwork adapter (EXPX9501AFXSR, Intel Corporation) for online datastreaming, a high-performance graphics adapter (Quadro FX5800, NvidiaCorporation) for GPU-based processing, and a server board (55520SC,Intel Corporation).

As mentioned above, the image data management unit 618 is connected tothe image acquisition workstation 610 and the image processingworkstation 620 by Gigabit optical fiber and includes a rack-mountserver with triple-channel SAS interface and a 10 Gigabit fiber networkadapter (EXPX9501AFXSR, Intel Corporation), as well as two 24-disk RAIDenclosures (ESDS A24S-G2130, Infortrend). The RAID enclosures areequipped with 48 SATA-2 hard disks (Ultrastar A7K2000, Hitachi), whichform two RAID-6 arrays for long-term imaging experiments andpost-acquisition wavelet-compressed data storage.

Procedure

Referring to FIG. 7, a procedure 700 is performed by the microscopesystem 100 to image the complex biological specimen 101. Initially, thebiological specimen 101 is prepared (702). The biological specimen 101is prepared (702) by chemically and biologically preparing the specimen,physically transferring or mounting the specimen to the holder 166, andplacing the holder 166 inside the chamber 168.

The microscope 110 is prepared (704), for example, by adjustingproperties (such as the alignment) of the light sheets 102, 104. Oncethe microscope 110 is prepared, the light sheets 102, 104 are generated(706). The lights sheets 102, 104 are directed through the biologicalspecimen 101 such that there is spatial and temporal overlap within thespecimen 101 (708). The beginning of the illumination and recording ofthe fluorescence can start at the moment the fertilized egg is formed,to enable imaging of the biological specimen 101 in its development froma fertilized egg to a complex system.

Fluorescence emitted from the biological specimen 101 is recorded by thecameras 106, 108 until the entire biological specimen 101 is captured(710). It is then determined if the imaging of the biological specimen101 should continue (712). For example, imaging can usually continueuntil the onset of strong muscle contractions in the developing embryo;at that point, imaging can be stopped because the specimen 101 becomesmore physically active and can be more difficult to image. However, itis possible that imaging could continue past this developmental point.

If imaging is to continue (712), then the relative alignment between thelight sheets and the biological specimen 101 is reset (714) and thefluorescence is once again recorded (710). The image of the biologicalspecimen 101 is created (716) and additional post processing can beperformed (718).

Next, a discussion of the preparation of the biological specimen 101(702) is provided with reference to FIG. 5B. As an example, the specimen101 can be microinjected 500 with some material such as drugs, labeledproteins (such as the fluorophores), or antibodies, as needed, and itcould also be dechorionated (that is, the chorion (or outermostmembrane) can be cleared or removed to enable subsequent imaging). Ifthe holder 166 includes a glass capillary 200, then the capillary 200can be pre-filled with a liquid 505 such as agarose gel, and then thespecimen 101 can be carefully transferred 510 into the capillary 200.

After transferring into the capillary 200, the specimen 101 can becentered and properly oriented within the agarose gel 505 that is withinthe capillary 200 so that one of its axes (such as itsanterior-posterior axis) is oriented parallel to the symmetry axis ofthe capillary 200. Moreover, during this step, the specimen 101 can bemaintained at a stable temperature. After the gel is allowed to settle,the section of the gel 505 that contains the specimen 101 can be forcedout 515 of the capillary 200 to provide full optical access to thespecimen 101, as shown in the illustration of FIGS. 5A and 5B. Thecapillary 200 with the specimen 101 embedded within the gel 505 can thenbe mounted 520 to the base 201 so that its symmetry axis aligns with thex axis of the microscope 110. And, the base 201 with the capillary 200and specimen 101 embedded within the gel 505 is mounted 525 within thechamber 168.

The microscope can be prepared 704 by adjusting one or more of thefollowing aspects of the microscope 110. For example, the displacementof each light sheet 102, 104 away from the y axis that passes throughthe mounted biological specimen 101 can be adjusted. The tilt of eachlight sheet 102, 104 relative to the y axis that passes through themounted biological specimen 101 can be adjusted. The relative intensityof the light sheets 102, 104 can be adjusted. One or more of the tiltand the displacement of each of the detection subsystems 116, 118relative to the z axis that passes through the mounted biologicalspecimen 101 can be adjusted. The relative detection efficiency of eachof the detection subsystems 116, 118 can be adjusted.

The light sheets 102, 104 are generated (706) by rapid laser scanningusing the optical scanner devices 140. The light sheets 102, 104 can begenerated from a continuous wave light sheet, which is shuttered so thateach light sheet arrives at a distinct point in time in the biologicalspecimen 101 but still within the temporal overlap. The light sheets102, 104 can be generated from pulsed wave light sheets, and can beactivated in synchrony.

For an implementation that uses one-photon excitation, the activation ofthe light sheets 102, 104 can be alternated in the two illuminationsubsystems 112, 114 for each z plane that is being imaged at any moment.For example, fast laser shutters can be used in both of the illuminationsubsystems 112, 114 to stagger the light sheets 102, 104 when usingcontinuous illumination with one-photon excitation.

For an implementation that uses two-photon excitation, the lightscattering in the illumination process is minimal, and fluorescenceexcitation is spatially confined to the image volume IV so that there islittle degradation in image quality in the biological specimen 101outside of the central region 103, and thus, the illumination subsystems112, 114 can be activated in synchrony. Moreover, it is possible toslightly displace the two focal volumes of the respective light sheets102, 104 just enough to obtain a continuous image without significantoverlap of the fluorescence contribution from each light sheet 102, 104.In contrast to one-photon excitation, there is no blurring of thefluorescence signal contribution along the illumination path, since thetwo-photon excitation is effectively suppressed once the light sheetbecomes too wide.

For one-photon excitation, a laser operating at 488 nm can be set to apower of a few hundred μW in the live imaging of Drosophila embryos.This corresponds to an exposure of the specimen to about 7-10 μJ lightenergy per acquired image pair, or about 1-3 mJ for the acquisition ofall images in a four-view image data set of the entire specimen perrecorded time point. For two-photon excitation, a laser power of about300 mW can be used to perform live imaging of Drosophila embryos at anexcitation wavelength of 940 nm.

The light sheets 102, 104 are directed through the biological specimen101 such that there is spatial and temporal overlap within the specimen101, as shown schematically in FIG. 1B. Initially, the light sheets 102,104 can be directed along a base x-y image plane (located at zA) asshown in FIG. 8A.

The fluorescence emitted from the biological specimen 101 is detected bythe cameras 106, 108 as images, and this image data is received by andproperly processed by the electronics controller 180, and then recordedwithin the computational system 190 (710).

The fluorescence at each x-y image plane of the specimen 101 is recordeduntil the entire specimen 101 is captured (710). For example, after thefluorescence at the zA image plane is recorded (see FIG. 8A), therelative placement along the z axis between the biological specimen 101and the light sheets 102, 104 is modified by a step along the z axis sothat the next x-y image plane can be recorded. In this way, thefluorescence emitted from the biological specimen 101 is recordedincrementally at each of the x-y image planes of the biological specimen101.

For example, in FIG. 8B, the fluorescence is recorded at the x-y imageplane located at zB; in FIG. 8C, the fluorescence emitted from thebiological specimen 101 is recorded at the x-y image plane located atzC; and in FIG. 8D, the fluorescence emitted from the biologicalspecimen 101 is recorded at the x-y image plane located at zD.

In some implementations, the relative placement between the biologicalspecimen 101 and the light sheets 102, 104 can be modified by moving thebiological specimen 101 along the z axis in steps while maintaining thelight sheet 102, 104 and the detection subsystems 116, 118 stationary.One advantage of this method is that it is possible to record largerimaging volumes since the linear motors that move the biologicalspecimen 101 have a longer travel range than the actuators that controlthe light sheet 102, 104, and the detection subsystems 116, 118.

In other implementations, the relative placement between the biologicalspecimen 101 and the light sheets 102, 104 can be modified by moving thelight sheets 102, 104 and the detection subsystems 116, 118 in synchronyalong the z axis in steps while maintaining the biological specimen 101stationary. One advantage of this faster imaging method is that it canbe faster to image because the actuators that control the light sheets102, 104 and detection subsystems 116, 118 are much faster than thelinear motors actuating the biological specimen 101. Another advantageof this imaging method is that the position of the light sheets 102, 104can be adjusted as it is moved to also provide optimal or improved imagequality at each step along the z axis and thus can provide animprovement in image quality across the specimen 101.

In some implementations in which the cameras 106, 108 are scientificCMOS (sCMOS), in order to acquire one image, the exposure time for thecamera 106, 108 can be set to 0.1-10 milliseconds (ms). In this case,because of the specific properties of the trigger mode for the sCMOScamera, typical image acquisition times have a range of about 10-30 ms,including the time for scanning the light sheets 102, 104 andtransferring the data of the images to the computational system 190 forprocessing.

In some implementations, both of the cameras 106, 108 are synchronized,that is, two images are recorded simultaneously within the acquisitiontime range. Thus, for the sCMOS cameras, it is possible to record at arate of about 2×100 images per second, with an image size of 2048×2048pixels, that is, 4 Megapixels.

Using the faster imaging method described above, it is possible torecord the entire three dimensional data set of a biological specimen101 having a thickness of about 200 μm (along the z axis) within a timeframe of 1-3 seconds (710).

In some implementations, the computational system 190 creates an imageof the biological specimen 101 (716) after the data acquisition of theimages of the entire biological specimen 101 has completed. As discussedabove, in some implementations, the data acquisition of the images canbe completed at the onset of strong muscle contractions in the embryo.However, it is possible to stop data acquisition earlier or later thanthis point in development, depending on what needs to be imaged.

In other implementations, the computational system 190 processes thedata during the acquisition of the data by the microscope 110. Forexample, it is possible that the computational system 190 could be setup to process all data recorded after the fluorescence from the entirebiological specimen 101 is captured at a particular x-y image plane(710) so that the computational system 190 is processing the data whilethe microscope 110 continues to record the fluorescence at the next x-yimage plane. Other setups for multitasking the creation of the imagewith the recording of the fluorescence are possible.

For simplicity, the following discussion assumes that in step 716, theimaging has completed and all data has been collected to create theimage of the biological specimen 101.

Referring also to FIG. 9, the microscope system 100 performs anexemplary procedure 716 for creating an image of the biological specimen101 at each of the x-y image planes that were acquired and recorded(710). For example, an image of the biological specimen 101 is createdat each of the x-y image planes corresponding to the exemplary values ofzA, zB, zC, and ZD shown, respectively, in FIGS. 8A-D.

Initially, one of the detection subsystems is set as the first view tobe imaged (900). For example, the detection subsystem 116 (as shown inFIG. 10A) can be set as the first view to be imaged.

Images of the biological specimen 101 that were recorded at this setview (the detection subsystem 116) are registered (902). This means thatthe images that are captured or recorded by the detection subsystem 116and stored within the computational system 190 are aligned with eachother. In particular, the image obtained from the illumination of thebiological specimen 101 with the light sheet 102 is aligned with theimage obtained from the illumination of the biological specimen 101 withthe light sheet 104.

In some implementation, registration at the set view can involve thefollowing procedure. Specifically, the image acquisition workstation 610performs data interpolation and calculates three-dimensional masks thatenvelope the recorded image of the biological specimen 101 obtained fromthe illumination with the light sheet 102 and the recorded image of thebiological specimen 101 obtained from the illumination with the lightsheet 104. A Gaussian filter can be used for envelope smoothening. Usingthe combined masks, the geometrical center of the biological specimen101 along the axis of the incident light sheets (the y axis) iscalculated as a function of the location in the x, z plane. Theresulting two-dimensional coordinate matrix indicates the y-centers ofthe optical illumination light path as a function of the location in thex, z plane in a first-order approximation. Using this matrix, slices ofabout 10 pixel thickness are extracted, background-corrected andregistered for both images. The optimal transformation settings aredetermined, considering sub-pixel-precision z-translation and y-rotationas the only degrees of freedom.

The choice of a data slice in the center of the optical light path forthe purpose of this first registration step is useful for thosebiological specimens 101 having bilateral optical symmetry with respectto the x, z plane (such as Drosophila and zebrafish embryos) and agenerally reasonable starting point in the absence of detailed knowledgeof the three-dimensional optical properties of the biological specimen101. The coordinate system of the recorded image obtained fromillumination with the light sheet 102 can constitute a reference orfirst channel; and the data of the image obtained from illumination withthe light sheet 104 (the second channel) can be transformed into thecoordinate system of the reference channel by global application of thetransformation parameters determined for the registration slice.

The image processing workstation 620 subsequently determines averageintensities in the registration slices and applies the correspondingintensity correction factor to the second channel.

Next, the registered images at the set view are fused (904) to form afused image at this set view; which means that the information contentof the registered images are combined into a single representation of animage. For example, if different parts or structures of a large objectare captured in the different registered images and these imagescomplement each other with respect to their information content, thenthe fusion of the images generates a single fused image that shows allof these parts or structures at the same time, and at the resolution orat the level of detail present in the contributing individual images.

In some implementation, fusion of the registered images at the set viewcan involve the following procedure. The image processing workstation620 fuses the two data sets by one of the following three methods:global 5-level wavelet decomposition with a Daubechies D4 basis (formaximum quality), linear blending in a 20-pixel transition regionrelative to the coordinates of the registration matrix (for large convexspecimens with complex optical properties), or global arithmeticaveraging (for maximum speed).

Next, it is determined whether there are any other views of thebiological specimen 101 that were imaged (906) and if there are, thenanother detection subsystem is set as the view to be imaged (908). Forexample, the detection subsystem 118 (as shown in FIG. 10B) can be setat step 908 as the view to be imaged next.

Images of the biological specimen 101 that were recorded at this setview (the detection subsystem 118) are registered (902); and theseregistered images at this set view are then fused (904) to form a fusedimage at this set view, as discussed above.

If it is determined that there are no other views taken of thebiological specimen 101 that were imaged (906), then the computationalsystem 190 registers the fused images (or fused data sets) that wereformed at each view (910). Thus, in the example above in which there areonly two views, the fused image obtained from the data taken by thedetection subsystem 116 is aligned with the fused image obtained fromthe data taken by the detection subsystem 118, as shown in FIG. 10C.

For example, the image acquisition workstation 610 calculates thethree-dimensional masks that envelope the recorded biological specimen101 in the two fused data sets, using a Gaussian filter for envelopesmoothening. Using the combined masks, the geometrical center of thespecimen 101 along the axis of the detection sub-systems (z-axis) iscalculated as a function of (x, y) location. The resultingtwo-dimensional coordinate matrix indicates the z-centers of the opticaldetection light path as a function of (x, y)-location in a first-orderapproximation. This coordinate matrix is then used to extract slices of10 pixel thickness from the fusion data sets. The optimal transformationsettings for registration of these two slices are determined,considering sub-pixel-precision x- and y-translations as well asz-rotation as degrees of freedom.

A region in the center of the optical detection light path can be usedfor the purpose of the second registration step, since the two detectionsubsystems 116, 118 typically provide comparable image quality in thislocation. The fused data set for the subsystem 116 constitutes thereference data set. The fused data set for the subsystem 118 istransformed into the coordinate system of the reference data set byglobal application of the transformation parameters determined for theregistration slice.

Next, the computational system 190 fuses these now aligned (registered)fused data sets into a final image representative of the image x-y planeof the biological specimen 101 that corresponds to the spatial andtemporal overlap of the light sheets 102, 104 within the specimen 101(912).

As an example, image processing workstation 620 operates on the aligned(registered) fused data sets for the two detection subsystems 116, 118,in direct analogy to the procedure described above for the fusing theregistered images at each of the subsystems 116, 118.

For example, as shown in FIG. 11, for fluorescence emitted from thebiological specimen 101 and recorded at the x-y image plane located at zequaling zC (as shown in FIG. 8C), a final image 1111 is created for theslice through the biological specimen 101 at the value of z=zC. Theimage of the biological specimen 101 is created for each x-y image planethat is imaged along the z steps. It is possible to create amaximum-intensity projection of all of these images to form a threedimensional image of the biological specimen 101.

As also shown in FIG. 11, a time series is taken at specific intervalsof time (t(i), t(i+100), t(i+200), t(i+300), t(i+400)), measured inarbitrary units, for the sub-region 1113 of the recorded image 1111.This time series shows the nuclear dynamics during one mitotic cycle(such as the 13^(th) mitotic cycle) of the specimen 101. The microscopesystem 100 is able to produce images like this that quantitativelyresolve the chromosomes within the cell nucleus across the entirebiological specimen 101. In this example, exemplary intervals of timeare shown, but many more intervals of time can be captured, as discussedin the examples below.

In summary, the procedure 700 provides for simultaneous four-viewimaging data. It employs multi-threading to take full advantage of thecomputational infrastructure within the computational system 190, andachieves an image data processing rate of 200 megabytes per second whenoperating in full-processing mode and a four times higher rate whenusing interpolated transformation parameters.

Computation of these transformation parameters and image fusion can beperformed by the workstations within the computational system 190 andcan be coded in any suitable programming language, such as, for example,Matlab and/or C++. This conceptual separation allows considering onlythe degrees of freedom relevant for the particular data type in therespective module and, thus, performing the multiview fusion moreefficiently.

As discussed above, post processing can be performed (718). Postprocessing can involve steps carried out by the computational system 190or can involve steps carried out by the operator.

For example, with reference to FIGS. 12A and 12B, the biologicalspecimen 101, which can be embedded in the agarose gel 505, can betransferred to a dissection microscope to control for normal hatching oflarvae 1201. The biological specimen 101 (which is now the larvae 1201)could be monitored to verify physiological development.

As another example, the computational system 190 could create a singleimage from the “stack” of two-dimensional images that are created; inwhich the stack of images corresponds to each image created for each zstep at step 912. The single image can be created by projecting themaximum pixel intensity levels through the entire stack onto a singleplane. This single image can be considered to be a two dimensionalvisualization of the three-dimensional data that is recorded with themicroscope system 100 and it can provide an overview of the recording.

EXAMPLES

Multiview Imaging of Drosophila Embryos

The microscope system 100 was used for performing simultaneous multiviewimaging of developing Drosophila embryos with sub-cellular resolution.

Referring to FIG. 13A, optical slices from a simultaneous multiview invivo recording of a nuclei-labeled stage 16 Drosophila embryo (dorsalside up) are obtained with one-photon excitation using the microscopesystem 100. A stage 16 embryo is an embryo that has undergone a certainnumber of development cycles (such as cell division or mitoses) and iscurrently in the middle of a particular development cycle. The slicesare taken along image planes at distinct values of z; for example, theimages that are shown in FIG. 13A are taken at the following values forz (starting from the top of the page) z=16.25 μm, 26.40 μm, 42.65 μm,54.84 μm, 77.18 μm, 91.40 μm, 123.89 μm, 148.26 μm, 158.42 μm, and176.70 μm. In all, many more images are recorded for a single specimen101. For example, about 100 images can be recorded in steps on the orderof a micrometer (for example, 2.03 μm). The depicted slices in FIG. 13Arepresent only a small subset of the total number of slices recordedacross the volume of the specimen 101.

FIG. 13B shows a maximum-intensity projection of a recording of a stage5 Drosophila embryo obtained with one-photon excitation using themicroscope system 100. The white rectangle indicates the regioncorresponding to the time series shown below the projection, and islocated in the transition region between two optical views. FIG. 13Cshows the time-series of the nuclear dynamics during the thirteenthmitotic cycle of the rectangular region of FIG. 13B. The nucleardynamics are quantitatively resolved in the entire embryo by one-photonexcitation using the microscope system 100. FIG. 13D showsmaximum-intensity projections of the images of the fluorescence obtainedduring one-photon excitation using the microscope system 100 on theDrosophila embryo. Separate projections are shown for dorsal and ventralhalves of the fused and background-corrected three-dimensional imagestacks. The embryo was recorded in 30-second intervals, using anacquisition period of 15 seconds per time point. The complete recordingcomprises one million images (11 terabytes) for about 2,000 time pointsrecorded from 3 to 18.5 hours post fertilization. The reference to PCindicates pole cells, the reference to VF indicates a ventral furrow,the reference to A indicates amnioserosa, and the reference to PSindicates posterior spiracles. The scale bars are shown for exampleonly.

Referring to FIG. 14A, optical images are taken with the microscopesystem 100 of in vivo recording of a nuclei-labeled stage 16 Drosophilaembryo (dorsal side up) using a two-photon excitation scheme. The slicesin this example are taken along image planes at distinct values of z;for example, the images of the embryo that are shown in FIG. 14A aretaken at the following values for z (starting from the top of the page)z=17.05 μm, 30.86 μm, 46.69 μm, 60.49 μm, 78.36 μm, 103.12 μm, 123.42μm, 140.48 μm, 152.66 μm, and 166.46 μm. In all, many more images can berecorded for a single specimen 101. For example, about 100 images can berecorded in steps on the order of a micrometer (for example, 2.03 μm).The depicted slices in FIG. 14A represent only a small subset of thetotal number of slices recorded across the volume of the specimen 101.

FIG. 14B shows the maximum-intensity projections of a time-lapserecording of Drosophila embryonic development taken with the microscopesystem 100 using a two-photon excitation scheme. Separate projectionsare shown for dorsal and ventral halves of the fused three-dimensionalimage stacks. The entire embryo was recorded in 30-second intervals overa period of 2 hours during germ band retraction, using an acquisitionperiod of 20 seconds per time point. The complete recording comprises37,620 high-resolution images (387 gigabytes). FIG. 14C shows themaximum-intensity projections of a time-lapse recording of Drosophilaembryonic development taken with the microscope system 100 using atwo-photon excitation scheme. The entire embryo was recorded in30-second intervals over a period of 3 hours during dorsal closure andventral nerve cord formation, using an image acquisition period of 20seconds per time point. The complete recording comprises 68,460high-resolution images (705 gigabytes). In the images, the term rGBrefers to a retracting germ band, the term A refers to amnioserosa, theterm VNC refers to a ventral nerve cord, and the term BL refers to brainlobes. Scale bars are shown for example only.

As evident from these images, the recording is free of the spatial andtemporal artifacts intrinsic to sequential multiview imaging, andprovides excellent temporal sampling of nuclei movements,characteristics that enable comprehensive nuclei tracking in thesyncytial blastoderm.

The one-photon excitation scheme (results of which are shown in FIGS.13A-D) provides an excellent signal-to-noise ratio for small penetrationdepths but was less able to capture structures deep inside the embryo.In contrast, the two-photon excitation scheme (results of which areshown in FIGS. 14A-C) has reduced autofluorescence when compared withthe one-photon excitation scheme and provided near complete physicalcoverage of the embryo even for late developmental stages. Thetwo-photon excitation scheme can require longer exposure times than theone-photon excitation scheme because of the potentially lower signalrate when using two-photon excitation.

A temporal resolution of 30 seconds for the whole embryo was obtainedusing the two-photon excitation scheme, and it captured global cellulardynamics during germ band refraction, dorsal closure, and ventral nervecord formation.

Cell Tracking in Drosophila Embryos

Cell tracking in entire developing Drosophila embryos has so far beentechnically challenging. Although there are impressive quantitativestudies of cell behavior, existing methods are limited to partialspatial observations and rely on time-consuming semi-automatedapproaches to image processing, which may be scaled to analyze the fullembryo.

Referring to FIGS. 15A-E, in addition to basic imaging, the microscopesystem 100 enables cell tracking in the entire Drosophila embryo. Asshown in FIG. 15A, global nuclei tracking in the entire Drosophilasyncytial blastoderm is produced from raw image data from a video thatwere superimposed with automated tracking results using a sequentialGaussian mixture model approach. Images show snapshots before thetwelfth mitotic wave and after the thirteenth mitotic wave, using arandom color scheme in the first time point, which is propagated todaughter nuclei using tracking information. As shown in FIG. 15B, globaldetection of nuclear divisions during the thirteenth mitotic wave in theDrosophila syncytial blastoderm is shown; non-dividing nuclei are shownin cyan and dividing nuclei are shown in magenta. The color of dividingnuclei progressively fades back to cyan within five time points. FIG.15C shows an enlarged view of a reconstructed embryo with nucleitracking information on the left and morphological nuclei segmentationon the right. The nuclei tracking information is generated from a randomcolor scheme (which is converted into a random pattern scheme in blackand white). FIG. 15D shows the average nucleus speed as a function oftime after nuclear division. Values at t=0 represent all pre-mitoticnuclei. Values at t greater than 0 represent post-mitotic nuclei at timet after mitosis. The small standard error of the mean (or s.e.m.) (equalto or smaller than line thickness) arises from the large specimen sizeof ˜2,500-5,000 specimens per time point. FIG. 15E shows thedistribution of the distances between nearest nuclei neighbors. Mean andstandard deviation of the post-mitotic distributions are 7.57±1.34 μm(12^(th) wave; n=1.44×10⁵) and 5.52±0.99 μm (13^(th) wave; n=4.66×10⁵).

By taking advantage of the improved spatio-temporal resolution,Drosophila embryo cell nuclei can be successfully reconstructed andtracked through multiple division cycles in the entire syncytialblastoderm using the microscope system 100. The nuclei can be trackedthrough the 12^(th) and 13^(th) mitotic waves, which represent some ofthe fastest global processes in the embryo. Nuclei were detectedautomatically with an accuracy of 94.74%±0.68% with respect to falsepositives (for those detected, but that are not present) and almost 100%with respect to false negatives (for those not detected), as evaluatedby the human expert. The segmentations were obtained using twoindependent methods: an efficient implementation of the Gaussian MixtureModel (GMM), which provides nuclear positions and size estimates (asshown in FIGS. 15A and B), and a three-dimensional implementation of thediffusion gradient vector field algorithm, which yields full nuclearmorphologies (as shown in FIG. 15C). The GMM-based segmentation andtracking was implemented on a GPU, which permitted reconstruction ofnuclei dynamics in the entire embryo in only 40 seconds per time point.

Owing to the high temporal resolution in the simultaneous multiviewlight sheet microscope system 100, it was sufficient to initialize eachtime point with the mixture model from the previous time point in orderto obtain tracking information. This approach yielded a trackingaccuracy between frames of 98.98%±0.42%. To follow nuclei through theirdivision, a machine learning classifier was trained based on local imagefeatures. This approach yielded a nuclei division detection and linkageaccuracy of 93.81%±2.71% throughout the recording. Three distinct typesof motion are quantified: global nuclei displacements, synchronizedwaves of nuclear division, and fast local nuclei displacements (daughternuclei separating after division).

The results are summarized in FIGS. 15D and E. The quantitative analysisof mitotic waves reveals that average nucleus movement speeds arehighest directly after nuclear division (8.12±2.59 μm/min in 12^(th)wave and 7.21±2.21 μm/min in 13^(th) wave, mean±s.d.; n=2,798 and 4,852,Huber robust estimator) and exhibit two pronounced local maxima at 2.1and 5.0 minutes after division (12^(th) wave; 3.8 and 6.3 min for13^(th) wave), which relate to the relaxation process of the globalnuclei population towards a new packing pattern (FIG. 15D). The averagedistance between daughter nuclei reaches a maximum 1.25 minutes (12^(th)wave) and 1.67 minutes (13^(th) wave) after division, which is almosttwo-fold higher than the global average nearest neighbor distance in theembryo (7.57±1.34 μm in 12^(th) wave and 5.52±0.99 μm in 13^(th) wave,mean±s.d.; n=1.44×10⁵ and 4.66×10⁵, Huber robust estimator), and relaxesto 8.76 μm for mitotic wave 12 and to 5.68 μm for mitotic wave 13, owingto the almost two-fold increase in nuclei count by the end of eachmitotic wave.

Determination of nuclei positions, nuclei sizes and nuclei tracking wereperformed within the computational system 190 by modeling each image asa mixture of Gaussians and sequentially estimating the mixtureparameters across time. Approximating nuclear shape intensity by aGaussian provides a good trade-off between model complexity and shapeinformation. In particular, each image can be modeled as:

${{I\left( {x,y,z} \right)} \propto {\sum\limits_{k = 1}^{K}\;{\pi_{k}{N\left( {{\left( {x,y,z} \right);\mu_{k}},\sum\limits_{k}^{\;}}\; \right)}}}},$where K is the number of objects in the image, μ_(k) is the centerlocation of each nucleus, Σ_(k) is the covariance matrix (representingthe shape of each nucleus as an ellipsoid), and π_(k) is the relativeintensity. Given each image as an input, these parameters can beestimated in a maximum-likelihood framework using, for example,Expectation-Maximization algorithm.

Due to the fine temporal resolution and excellent spatial coverageachieved by the simultaneous multiview light-sheet microscope system100, we used each solution from time T_(i-1) as an initialization fortime point T_(i). Given that each Gaussian in an image derives from aGaussian in the previous time point, tracking information can bedirectly recovered. To handle cell divisions, a set of examples withcells dividing and cells not dividing is collected, a machine learningclassifier is trained based on local image features.

The tracking using Gaussian mixture models was implemented on a graphicsprocessing unit (GPU) and can be implemented using CUDA™, whichincreases computing performance (and can result in a 100× speed-up),which permits the processing of nuclear positions and movements in theentire embryo with thousands of nuclei and millions of voxels in only 40seconds per time point, using a processing workstation with a singleTesla GPU (Nvidia Corporation).

Drosophila Neural Development

Referring to FIGS. 16A-G, the microscope system 100 is used toinvestigate neuron type specification and axonal guidance in theDrosophila nervous system. Neuro-developmental dynamics are oftenstudied by comparing fixed specimens at different developmental stages,but also by live imaging in a local context. The microscope system 100can record and image data on the developmental dynamics of the entireembryonic nervous system.

FIG. 16A shows the results of imaging using the microscope system 100 ina one-photon excitation scheme of a histone-labeled Drosophila embryo asthe specimen 101, superimposed with manually reconstructed lineages ofthree neuroblasts and one epidermoblast for 120-353 minutes postfertilization (time points 0-400). The track color shown encodes timeand the scale bar corresponds to 30 μm. FIG. 16B shows the enlarged viewof the tracks highlighted in FIG. 16A. The green spheres show celllocations at the time point 400. The asterisks mark six ganglion mothercells produced in two rounds of neuroblast division. The term NB refersto a neuroblast and the term EB refers to an epidermoblast.

FIGS. 16C-E show maximum-intensity projections (dorsal and ventralhalves) of Drosophila embryonic nervous system development, recordedwith the microscope system using a one-photon excitation scheme. TheDrosophila (for example, elav(C155)-GAL4,UAS-mCD8::GFP transgenic)embryo was recorded in 30-second intervals over the period 9.5-15.3hours post fertilization (about 700 time points), using an imageacquisition period of 15 seconds per time point. The intensitynormalization was performed within the computational system 190 tocompensate for GFP signal increase over time. The autofluorescentvitelline membrane was computationally removed in FIG. 16C.

FIG. 16F shows an enlarged view of the area highlighted in FIG. 16E.FIG. 16G shows a progression of maximum-intensity projections of axonalmorphogenesis in a Ftz-ng-GAL4,10XUAS-IVS-myr::GFP transgenic embryo(false color look-up-table), recorded with the microscope system 100using a one-photon excitation scheme. The images represent a 0.3%sub-region of the total covered volume. The term VNC refers to theventral nerve cord, the term PNS refers to the peripheral nervoussystem, the term B refers to the brain, the term VM refers to thevitelline membrane, the term EID refers to the eye-antennal imaginaldisc, the term VSC refers to the ventral sensory cells, the term LSCrefers to the lateral sensory cells, the term DSC refers to the dorsalsensory cells, the term A refers to the antenna, the term BC refers tobrain commissure. The asterisk indicates shortening of the VNC. Thescale bars that are shown are provided as examples, and are as follows:In FIGS. 16A and 16B, the bars are 30 μm in length; in FIGS. 16C-E, thebars are 50 μm in length; in FIG. 16F, the bar is 10 μm in length; andin FIG. 16G, the bar is 5 μm.

Transgenic embryos expressing elav(C155)-GAL4 and UAS-mCD8::GFP wereused to visualize all post-mitotic neurons, and time-lapse one-photonimaging was performed of the entire embryonic nervous system with 25 and30 seconds time resolution, capturing neural development in about400,000 high-resolution images (4 terabytes) for more than 700 timepoints. Recording a four-view data set of the entire embryo requiredonly 2 mJ of light energy per time point and showed negligiblephoto-bleaching. The resulting data sets provide detailed information onthe development of the central and peripheral nervous systems (See FIGS.16C-F) and reveal fine details in the dynamics of axonal outgrowth. Forinstance, the time-lapse recordings show several clusters of abdominalsensory organs (dorsal, lateral and ventral) and the formation ofconnectives in the ventral nerve cord. Strikingly, the dynamics of thesecells across different segments appear to be highly stereotypic,indicating the role of global cues. The imaging also captures thedynamics of sensory cells in the head region and in the posteriorsegments of the specimen 101. The entire morphogenesis of theeye-antennal imaginal disc can be followed, which separates from theembryonic brain to move in an anterior-medial direction. The antennaanlage separates from the larval eye (Bolwig's organ) as both establishtheir final locations. It is even possible to follow the morphogenesisof the deeply embedded embryonic brain, which starts as two bilaterallysymmetrical neurogenic regions, initially separated from each other, andbegins to move posteriorly with head involution. The development ofcommissures is visible at high temporal resolution.

Additionally, the microscope system 100 performed high-resolutionexperiments with Ftz-ng-GAL4,10XUAS-IVS-myr::GFP embryos exhibitingsparse expression in the central nervous system. The spatial sampling isincreased by more than a factor of six, maintaining the same hightemporal resolution and imaging for the same period of time (30-secondintervals for 8.5 hours, providing approximately 460,000 images or 4.6terabytes of data. These recordings preserve the capability to followglobal processes and at the same time reveal detailed filopodialdynamics during axonal morphogenesis at excellent spatio-temporalresolution (See FIG. 16G). For example, the high-speed imaging resultsshow that zones of filopodial extension and exploration are retainedproximal from the growth cones for long periods of time.

Additionally, to complement the automated reconstruction of cellulardynamics in early Drosophila embryos, the microscope system 100 can beused for cell tracking and cell lineage analyses in later developmentalstages of the specimen 101. For example, using a two-photon excitationscheme, manual cell tracking can be performed in non-superficial layersof the retracting germ band. In addition, neuroblast and epidermoblastlineages can be followed in a one-photon excitation scheme. The highsignal-to-noise ratio of the one-photon data permitted the tracking ofblastoderm cells through gastrulation and subsequent cell divisions fora total period of 400 time points (approximately 4 hours; as shown inFIGS. 16A and B). These reconstructions capture cellular dynamics in theearly stages of nervous system development, including neuroblastdelamination and birth of the first ganglion mother cells.

The computational system 190 performs specific actions to enable theacquisition of the morphologies of the nuclei within the specimen 101.In particular, an image I(x,y,z) that is created by the computationsystem 190 is segmented by simulating the fluorescence intensities asattractive forces (f) embedded in a medium governed by fluid flowequations. A gradient vector field v(x,y,z)=[u(x,y,z), v(x,y,z),w(x,y,z)] can be defined as the field that minimizes the functional,ε=∫∫∫μ(u _(x) ² +u _(y) ² +u _(z) ² +v _(x) ² +v _(y) ² +v _(z) ² +w_(x) ² +w _(y) ² +w _(z) ²)+|∇f| ² |v−∇f| ² dxdydz  (1)where,f=∇|G _(σ)(x,y,z)*I(x,y,z)|²  (2)and where G_(σ)(x,y,z) represents a three-dimensional Gaussian withstandard deviation σ and * denotes convolution. Briefly, using thecalculus of variations we obtain a set of Euler equations, which can besolved by treating u, v and w as functions of time,u _(t)(x,y,z,t)=μ∇² u(x,y,z,t)−(u(x,y,z,t)−f _(x)(x,y,z))×(f_(x)(x,y,z)² +f _(y)(x,y,z)² +f _(z)(x,y,z)²)  (3)v _(t)(x,y,z,t)=μ∇² v(x,y,z,t)−(v(x,y,z,t)−f _(y)(x,y,z))×(f_(x)(x,y,z)² +f _(y)(x,y,z)² +f _(z)(x,y,z)²)  (4)w _(t)(x,y,z,t)=μ∇² w(x,y,z,t)−(w(x,y,z,t)−f _(z)(x,y,z))×(f_(x)(x,y,z)² +f _(y)(x,y,z)² +f _(z)(x,y,z)²)  (5)

The steady-state solution of these linear parabolic equations is thesolution of the Euler equations and yields the required flow field. Theregularization parameter μ balances the diffusive (first term) vs. theadvective (second term) components in Eq. (1), that is, it determinesthe amount of smoothing exerted by the algorithm, and should beincreased in the presence of noise.

The gradient field calculation is followed by gradient flow tracking, inwhich similar groups of voxels were identified as those that “flow”towards the same sink. This generates a “mosaic” image, in which eachtile (the basin of one sink) contains only one object. The segmentationwas finalized by adaptively thresholding each tile. The computationalsystem 190 calculates the optimum threshold separating the two classes,fore- and background, so that their combined intra-class variance isminimal (Otsu's method). Voxels with values smaller than this thresholdwere considered background.

The high temporal resolution of the recordings enabled theinitialization of the diffusion gradient vector field segmentationalgorithms, for a time point T_(i), with the solution that converged inT_(i-1). This resulted in more than 20× convergence speed-up, and wasessential to obtain high-quality shape information and making thealgorithm practical for large 4D datasets.

The data sets that are stored within the computational system 190 andprocessed to create the image (716) are large in size (for example, theycan typically in the order of up to dozens of terabytes). When routinelyworking with these data sets, the operator of the microscope system 100can select spatial and temporal subsets of the images, and/or only imageframes that originate from a specific color channel, view angle,specimen, or camera 106, 108. If this is done, then the computationalsystem 190 can store and run a separate program that facilitates theorganization, browsing, and processing of the images in a way that isconvenient, fast, extensible, and with minimal network and I/O load. Tothis end, such a computer program can implement the following concepts:(a) virtual folders; by which the user can make sophisticated datasubset selections, (b) processing task definition is separated from dataselection (for re-use and documentation), (c) a plug-in system, for theeasy implementation of custom processing code, (d) connection toexternal programs Vaa3D and ImageJ, and (e) a convenient graphical userinterface for defining the virtual folders and processing tasks, as wellas for browsing through data sets (for viewing the images and theirassociated meta information).

In summary, a complete technology framework has been described thatprovides for light sheet-based one-photon and multi-photon simultaneousmultiview imaging and image analysis, which overcomes the limitations ofsequential multiview strategies and enables quantitative systems-levelimaging of fast dynamic events in large living specimens. The frameworkprovides near-complete physical coverage through the acquisition of aplurality of complementary optical views with a maximum time shift of 20milliseconds, independent of specimen size. Temporal correspondence ofcomplementary views is improved by more than three orders of magnitudeand imaging speed is improved more than 20-fold over light sheetmicroscopy with sequential four-view imaging. The system is designed forhigh-speed long-term imaging under physiological conditions, usingreal-time electronics and an advanced computational infrastructure forsustained data acquisition at 350 megabytes s⁻¹ (175 million voxels persecond) for up to several days. The system includes computationalsolutions for high-throughput multiview image fusion and image datamanagement of experiments, which typically comprise millions ofhigh-resolution images and up to several dozens of terabytes perspecimen.

In other implementations, the light sheets 102, 104 produced within theillumination subsystems 112, 114 can be produced with a staticarrangement and thus can be static light sheets. For example, theoptical components 132, 134 can include a telescope lens pair and acylindrical lens that focuses the light along only one direction.

The light sources 122, 124 can include a pulsed Ti:Sapphire laser (suchas a Chameleon Ultra II by Coherent, Inc. of Santa Clara, Calif.). Anadditional module can be placed between the light sources 122, 124 andthe respective optical component set 132, 134. The additional module canmodify the output of the light sources 122, 124 and can provide laserintensity modulation and IR beam splitting. The module consists of abeam attenuation sub-module (for example, an AHWP05M-980 mountedachromatic half-wave plate and GL10-B Glan-laser polarizer by ThorlabsInc. of Newton, N.J.), a Pockels cell with driver (such as a Model350-80-LA-02 KD*P series electro-optic modulator and a Model 302RMdriver by Conoptics Inc. of Danbury, Conn.) and an IR beam splitter(such as a broadband polarizing cube beam splitter such as modelPBSH-450-2000-100 by Melles Griot and an achromatic ½ wave plate such asmodel WPA1312-2-700-1000 by Casix, Inc. of China).

Switching Illumination and Detection Arms

Referring to FIG. 17, a microscope system 1700 is used to image a livebiological specimen 1701. The microscope system 1700 includes an opticalmicroscope 1710 having two or more optical arms (for example, opticalarms 1720 and 1740), with each arm including both an illuminationsubsystem (which is also referred to as a light source unit) and adetection subsystem (which is also referred to as a detection unit), andat least two of the optical arms being arranged along non-parallel (forexample, perpendicular) optical axes. For example, the optical arm 1720includes a light source unit 1722 and a detection unit 1724. At leastpart of the light source unit overlaps with and is shared by thedetection unit of that optical arm. The axis of an optical arm is givenby the path of the illumination light as it passes through theoverlapping or shared region of the light source unit and the detectionunit. An optical arm axis is non-parallel with another optical arm axisif they intersect at some point. When properly set up for datacollection, the at least two optical arm axes intersect within thebiological specimen 1701.

Each optical arm includes an optical data separation apparatus (forexample, apparatuses 1723 and 1743 of optical arms 1720 and 1740,respectively) on the path between the biological specimen 1701 and thelight source unit and the detection unit, the optical data separationapparatus being configured to separate optical data between the lightsource unit and the detection unit to enable each optical arm to haveboth illumination and detection functionality. The optical dataseparation apparatus separates the overlapping region of the lightsource unit and the detection unit with the non-overlapping regions ofthese units.

The microscope system 1700 enables recording of a three-dimensional dataset along two or more perpendicular directions without having to turnthe biological specimen between data acquisitions. The microscope system1700 provides uniform illumination across the entire depth of theilluminated plane of the biological specimen. In this way, additionaloptical views are obtained by switching the functionality between theillumination subsystems and the detection subsystems within each opticalarm. Such a design can provide for isotropic resolution of thebiological specimen without changing the point spread function of themicroscope system 1700.

The microscope system 1700 includes a specimen region 1799 configured toreceive the biological specimen 1701. The microscope system 1700includes at least first and second optical arms 1720 and 1740, with eachoptical arm having an optical path that crosses the specimen region1799, the optical path defining the optical arm axis. Each optical armincludes both a light source unit and a detection unit, and each opticalarm is arranged along a respective distinct arm axis.

In the design of FIG. 17, the optical microscope 1710 includes the firstoptical arm 1720 and the second optical arm 1740, with the respectivearm axes of the first and second optical arms 1720, 1740 being notparallel with each other (and, in this example, they are perpendicularto each other and meet up inside the biological specimen 1701). Thefirst optical arm 1720 includes a light source unit 1722 and a detectionunit 1724, and an optical path that crosses the specimen region 1799along a first arm axis 1721. The second optical arm 1740 includes alight source unit 1742 and a detection unit 1744, and an optical paththat crosses the specimen region 1799 along a second arm axis 1741.

The light source unit 1722 includes a light source and a set ofillumination optical devices arranged to produce and direct a firstlight sheet 1702A toward the specimen region 1799 along an illuminationaxis that is parallel with the first arm axis 1721 along the directiontoward the specimen region 1799. The light source unit 1722 alsoincludes a set of actuators coupled to one or more illumination opticaldevices.

The light source unit 1742 includes a light source and a set ofillumination optical devices arranged to produce and direct a secondlight sheet 1702B toward the specimen region 1799 along an illuminationaxis that is parallel with the second arm axis 1741 along the directiontoward the specimen region 1799. The light source unit 1742 alsoincludes a set of actuators coupled to one or more illumination opticaldevices.

The detection unit 1724 includes a camera and a set of detection opticaldevices arranged to collect and record images of fluorescence F2 emittedfrom the biological specimen 1701 received in the specimen region 1799along a detection axis that is perpendicular to the illumination axis ofthe second optical arm 1740. Because the illumination axis of the secondoptical arm 1740 is parallel with the second arm axis 1741, thedetection axis of the first optical arm 1720 (and the detection unit1724) is also perpendicular to the second arm axis 1741. The detectionunit 1724 also includes a set of actuators coupled to one or more of thecamera and the detection optical devices. The fluorescence F2 emittedfrom the biological specimen 1701 and collected by the detection unit1724 of the first optical arm 1720 arises at least in part due to theinteraction of the second light sheet 1702B (emitted from the secondoptical arm 1740) with the biological specimen 1701. In particular, thesecond light sheet 1702B excites fluorophores within the specimen 1701into higher energy levels, which then results in the subsequent emissionof a fluorescence photon P, and the fluorescence photons P (denoted asthe fluorescence F2) are detected by the detection unit 1724 of thefirst optical arm 1720.

The detection unit 1744 includes a camera and a set of detection opticaldevices arranged to collect and record images of fluorescence F4 emittedfrom the biological specimen 1701 received in the specimen region 1799along a detection axis that is perpendicular to the illumination axis ofthe first optical arm 1720. Because the illumination axis of the firstoptical arm 1720 is parallel with the first arm axis 1721, the detectionaxis of the second optical arm 1740 (and the detection unit 1744) isalso perpendicular to the first arm axis 1721. The detection unit 1744also includes a set of actuators coupled to one or more of the cameraand the detection optical devices. The fluorescence F4 emitted from thebiological specimen 1701 and collected by the detection unit 1744 of thesecond optical arm 1740 arises at least in part due to the interactionof the first light sheet 1702A (emitted from the first optical arm 1720)with the biological specimen 1701. In particular, the first light sheet1702A excites fluorophores within the specimen 1701 into higher energylevels, which then results in the subsequent emission of a fluorescencephoton P, and the fluorescence photons P (denoted as the fluorescenceF4) are detected by the detection unit 1744 of the second optical arm1740.

The first optical arm 1720 includes the optical data separationapparatus 1723 placed on the path between the specimen region 1799 andat least a portion of the light source unit 1722 and at least a portionof the detection unit 1724. The optical data separation apparatus 1723separates optical data between the light source unit 1722 and thedetection unit 1724. For example, the optical data separation apparatus1723 separates the first light sheet 1702A and the fluorescence F2within the components of the light source unit 1722 and the detectionunit 1724. Thus, the optical data separation apparatus 1723 ensures thatthe fluorescence F2 is directed toward the camera of the detection unit1724 but not toward the light source of the light source unit 1722 andthe optical data separation apparatus 1723 ensures that the first lightsheet 1702A is directed from the light source of the light source unit1722 toward the specimen region 1799 without being directed into thedetection unit 1724. The optical data separation apparatus 1723 can alsobe configured to block light sheets (such as the light sheet 1702B) fromother optical arms from entering the detection unit 1724.

The second optical arm 1740 also includes an optical data separationapparatus 1743 placed on the path between the specimen region 1799 andat least a portion of the light source unit 1742 and at least a portionof the detection unit 1744. The optical data separation apparatus 1743separates optical data between the light source unit 1742 and thedetection unit 1744. For example, the optical data separation apparatus1743 separates the second light sheet 1702B and the fluorescence F4within the components of the light source unit 1742 and the detectionunit 1744. Thus, the optical data separation apparatus 1743 ensures thatthe fluorescence F4 is directed toward the camera of the detection unit1744 but not toward the light source of the light source unit 1742 andthe optical data separation apparatus 1743 ensures that the second lightsheet 1702B is directed from the light source of the light source unit1742 toward the specimen region 1799 without being directed into thedetection unit 1744. The optical data separation apparatus 1743 can alsobe configured to block light sheets 1702 from other optical arms fromentering the detection unit 1744.

The optical data separation apparatus 1723, 1743 is at the location atwhich the optical paths of the two units (the light source unit 1722,1742 and detection unit 1724, 1744) meet. The optical data separationapparatus 1723, 1743 can be a color-separation device such as a dichroicmirror or beam splitter. The optical data separation apparatus 1723,1743 can be configured to transmit the illuminating laser light sheet(which can be at a relatively shorter wavelength) and reflect theemitted fluorescence light (which can be at a relatively longerwavelength). In other implementations, the illuminating laser lightsheet can be at a relatively longer wavelength than the wavelength ofthe emitted fluorescence light.

The microscope system 1700 also includes a translation system 1770coupled to one or more of the specimen region 1799, the light sourceunits 1722, 1742, and the detection units 1724, 1744, and configured totranslate one or more of the specimen region 1799, the light sheets1702A, 1702B produced by the light source units 1722, 1742, and thedetection units 1724, 1744 relative to each other along a linear axiswithout rotating the biological specimen 1701 received in the specimenregion 1799.

Though not required for imaging of many biological specimens, themicroscope system 1700 can also include a third optical arm 1730 and afourth optical arm 1750. A brief description of the third and fourthoptical arms 1730, 1750 is provided next, and the remainingimplementations shown and described include all four optical arms 1720,1730, 1740, 1750, even though only two optical arms (such as 1720 and1740) are adequate for imaging many types of biological specimens.

The third optical arm 1730 is arranged opposite to the first optical arm1720 so that the illumination axis of the third optical arm 1730 isparallel with the illumination axis of the first optical arm 1720, andis therefore parallel with the first arm axis 1721. The third opticalarm 1730 includes a light source unit 1732 and a detection unit 1734,and an optical path that crosses the specimen region 1799 along thefirst arm axis 1721. The fourth optical arm 1750 is arranged opposite tothe second optical arm 1740 so that the illumination axis of the fourthoptical arm 1750 is parallel with the illumination axis of the secondoptical arm 1740, and is therefore parallel with the second arm axis1741. The fourth optical arm 1750 includes a light source unit 1752 anda detection unit 1754, and an optical path that crosses the specimenregion 1799 along the second arm axis 1741.

The light source unit 1732 includes a light source and a set ofillumination optical devices arranged to produce and direct a thirdlight sheet 1704A toward the specimen region 1799 along an illuminationaxis that is parallel with the first arm axis 1721 along the directiontoward the specimen region 1799. The light source unit 1732 alsoincludes a set of actuators coupled to one or more illumination opticaldevices.

The light source unit 1752 includes a light source and a set ofillumination optical devices arranged to produce and direct a fourthlight sheet 1704B toward the specimen region 1799 along an illuminationaxis that is parallel with the second arm axis 1741 along the directiontoward the specimen region 1799. The light source unit 1752 alsoincludes a set of actuators coupled to one or more illumination opticaldevices.

For small, transparent samples, it is possible to obtain full physicalcoverage of the biological specimen 1701 using only the first lightsheet 1702A from the first optical arm 1720 and the second light sheet1702B from the second optical arm 1740. Due to light scattering and/orlight absorption inside the biological specimen 1701, larger,non-transparent biological specimens 1701 can have spatial regions thatare not easily reached by the first and second light sheets 1702A,1702B. The third optical arm 1730 and the fourth optical arm 1750provide additional information (such as higher contrast and higherresolution) for such larger and/or non-transparent specimens.

The detection unit 1734 includes a camera and a set of detection opticaldevices arranged to collect and record images of fluorescence F3 emittedfrom the biological specimen 1701 received in the specimen region 1799along a detection axis that is perpendicular to the illumination axis ofthe second optical arm 1740 and the fourth optical arm 1750. Because theillumination axis of the second optical arm 1740 and the fourth opticalarm 1750 is parallel with the second arm axis 1741, the detection axisof the third optical arm 1730 (and the detection unit 1734) is alsoperpendicular to the second arm axis 1741. The detection unit 1734 alsoincludes a set of actuators coupled to one or more of the camera and thedetection optical devices. The fluorescence F3 emitted from thebiological specimen 1701 and collected by the detection unit 1734 of thethird optical arm 1730 arises at least in part due to the interaction ofthe second light sheet 1702B (emitted from the second optical arm 1740)with the biological specimen 1701, or the interaction of the fourthlight sheet 1704B (emitted from the fourth optical arm 1750) with thebiological specimen 1701.

The detection unit 1754 includes a camera and a set of detection opticaldevices arranged to collect and record images of fluorescence F5 emittedfrom the biological specimen 1701 received in the specimen region 1799along a detection axis that is perpendicular to the illumination axis ofthe first optical arm 1720 and the third optical arm 1730. Because theillumination axis of the first optical arm 1720 the third optical arm1730 is parallel with the first arm axis 1721, the detection axis of thefourth optical arm 1750 (and the detection unit 1754) is alsoperpendicular to the first arm axis 1721. The detection unit 1754 alsoincludes a set of actuators coupled to one or more of the camera and thedetection optical devices. The fluorescence F5 emitted from thebiological specimen 1701 and collected by the detection unit 1754 of thefourth optical arm 1750 arises at least in part due to the interactionof the first light sheet 1702A (emitted from the first optical arm 1720)with the biological specimen 1701 or the interaction of the third lightsheet 1704A (emitted from the third optical arm 1730) with thebiological specimen 1701.

When additional optical arms 1730, 1750 are included in the microscopesystem 1700, the respective detection units 1724, 1744 of the first andsecond optical arms 1720, 1740 are configured to detect fluorescenceemitted from the respective additional interactions between the lightsheets 1704A and 1704B and the biological specimen, as follows. Inparticular, the detection unit 1724 of the first optical arm 1720collects the fluorescence F2 emitted from the biological specimen 1701,and the fluorescence F2 can also arise at least in part due to theinteraction of the fourth light sheet 1704B (emitted from the fourthoptical arm 1750) with the biological specimen 1701. Additionally, thedetection unit 1744 of the second optical arm 1740 collects thefluorescence F4 emitted from the biological specimen, and thefluorescence F4 can also arise at least in part due to the interactionof the third light sheet 1704A (emitted from the third optical arm 1730)with the biological specimen 1701.

Referring also to FIG. 21A, the use of the third light sheet 1704A,which is directed along the first optical axis 1721 opposite to thedirection along which the first light sheet 1702A is directed, allowsfor the displacement of the two light sheets 1702A and 1704A slightlywith respect to each other along the illumination axis (or the firstoptical axis 1721), such that these two light sheets 1702A, 1704A reachthe respective beam waist 2105, 2110 (or focal plane) at different axialpositions along the biological specimen, as also discussed above withreference to FIG. 1B. The beam waist is the beam width or diameter atthe minimum value. This strategy allows us to cover the entirefield-of-view with thinner light sheets than what would be possible witha single light sheet. In particular, the distance over which a focusedGaussian beam is sufficiently homogeneous in its diameter isproportional to the square of this diameter, and thus subdividing thefield-of-view into two parts that are each optimally covered by onelight sheet permits a reduction in the beam width of each light sheet bya square root of two compared to using a single sheet.

In summary, the microscope system 1700 is set up such that each opticalarm 1720, 1730, 1740, 1750 is equipped with both a light source unit anda detection unit, and at least two optical arms 1720, 1740, for example,are arranged perpendicularly to each other. The microscope system 1700records the fluorescence emitted from the biological specimen 1701 overits entire volume. The fluorescence is emitted from the labels withinthe biological specimen 1701 that include the fluorophores that aretuned to the wavelength of the light sheets directed to the biologicalspecimen 1701. In this way, biological structures, such as cells (forexample, neurons), within the biological specimen 1701 can be tracked(that is, traced or followed) as the biological specimen 1701 developsover time from a first stage (such as a single cell) to a second stage(such as an embryo). The acquired images can be fused together to obtaina three-dimensional data set approaching isotropic three-dimensionalresolution.

The microscope system 1700 includes a dual illumination/detectionconfiguration that translates the light sheets and objectives asdiscussed in greater detail below (relative to the biological specimen)along the detection axes that are parallel with the second optical axis1741 and perpendicular to the first optical axis 1721. For example, insome implementations as shown in FIG. 21A, the light source units 1722,1732 can produce and direct the respective light sheets 1702A, 1704A tothe biological specimen 1701 and the detection units 1744, 1754 detectthe emitted respective fluorescence F4, F5 from the biological specimen1701. The microscope system 1700 is set up to switch off the active pairof light source units (units 1722, 1732) and to activate the other pairlight source units 1742, 1752 (as shown in FIG. 21B) within the otheroptical arms 1740, 1750; and to switch off the active pair of detectionunits (units 1744, 1754) and to activate the other pair of detectionunits 1724, 1734 (as shown in FIG. 21B). Then, as shown in FIG. 21B, thedetection units 1724, 1734 record the respective fluorescence F2, F3emitted from the biological specimen 1701 over its entire volume asecond time, by translating the light sheets 1702B, 1704B and objectivesas discussed below (relative to the biological specimen 1701) along thedetection axes that are parallel with the first optical axis 1721 (andalso perpendicular to the previously-scanned detection axis that isparallel with the second optical axis 1741).

Referring to FIGS. 18 and 19, an exemplary implementation of amicroscope system 1800 including an optical microscope 1810 based on theconcepts of the optical microscope 1710 is shown and will be describednext.

The optical microscope 1810 includes four optical arms 1820, 1830, 1840,1850, each optical arm arranged along a respective arm axis and eachoptical arm 1820, 1830, 1840, 1850 including respective light sourceunits 1822, 1832, 1842, 1852 and respective detection units 1824, 1834,1844, 1853. A detailed description of the optical arms 1820 and 1840 areprovided next.

The first optical arm 1820 is arranged along a first arm axis 1821 andthe second optical arm 1840 is arranged along a second arm axis 1841.The first arm axis 1821 is parallel with the y axis and the second armaxis 1841 is parallel with the z coordinate of a Cartesian coordinatesystem.

The light source unit 1822 of the first optical arm 1820 includes alight source 1960, an optical control system 1961 that can include anillumination shutter that enables selective activation of the lightsheet 1802A and an illumination filter that blocks light outside of atarget wavelength range, a scanning arrangement 1962 that includes anoptical scanner device (such as one or more rotatable or movablemirrors) 1963 (shown in FIG. 19) and an f-theta lens 1964 (shown in FIG.19). One of the movable mirrors of the optical scanner device 1963 canbe used to scan the light beam from the light source 1960 along the xaxis to produce the light sheet 1802A while the other of the movablemirrors of the optical scanner device 1963 can be used to scan the lightsheet 1802A along a direction perpendicular or lateral to the x axis.The light sheet 1802A produced by the light source unit 1822 is producedby scanning an output of the light source 1960. The light beam from thelight source 1960 is directed to the optical scanner device 1963, whichdeflects the light beam along the x axis (which is out of the page inFIG. 18) to form a light sheet, which is directed toward the f-thetalens 1964. The optical scanner device 1963 deflects the incident laserlight to produce an angular range that defines the height of the lightsheet 1802A along the x axis in the specimen 1801. And, the angle of thelight beam that exits the optical scanner device 1963 is converted intoa displacement along the x axis by the f-theta lens 1964 because theoptical scanner device 1963 is positioned to be located at the focalplane of the f-theta lens 1964, thus producing a macroscopic lightsheet. Additionally, at the end of the x axis scan, the optical scannerdevice 1963 steps the light beam from the light source 1960 laterally(along a direction perpendicular to the x axis) so that the light sheet1802A is displaced along the z direction to illuminate an adjacent x-yimage plane in the biological specimen 1801.

The light source unit 1822 of the first optical arm 1820 shares at leastone component with the detection unit 1824 of the first optical arm1820. These shared components are a tube lens 1965 and a sharedobjective 1966. The shared components of the light source unit 1822 andthe detection unit 1824 are separated from the non-shared components ofthe respective light source unit 1822 and the detection unit 1824 by theoptical data separation apparatus, which, in this example, is a dichroicoptical element such as a dichroic beam splitter 1843.

When passing the light sheet 1802A from the non-shared components of thelight source unit 1822 toward the biological specimen 1801, the pair ofthe tube lens 1965 and the shared objective 1966 focuses the macroscopiclight sheet output from the f-theta lens 1964 into the microscopic lightsheet 1802A at the specimen 1801. The shared objective 1966 can be amicroscope objective that includes a single lens, or a combination oflenses and other optical elements.

The detection unit 1824 of the first optical arm 1820 includes a filter1968, and a camera 1969. Thus, the fluorescence F2 emitted from thespecimen 1801 is collected by the shared objective 1966, and focusedusing the tube lens 1965 onto the camera 1969. The filter 1968 rejectslight of wavelengths outside a wavelength band centered around thewavelength of the fluorescence F2 to be detected. Moreover, the filter1968 can be mounted on a filter wheel with other filters of distinctwavelength ranges so that the wavelength of the rejected light can bechanged by selecting a different filter on the wheel. The filter 1968can be a short-pass, a long-pass, or a band-pass filter, depending onthe type of excitation applied to the fluorophores. For example, forone-photon excitation of the fluorophores, the wavelength of the lightsheets is generally shorter than the wavelength of the fluorescence tobe detected, and thus, the dichroic beam splitter 1823 is set up totransmit the longer wavelengths to the camera 1969, and thus, the filter1968 can be a long-pass filter or a band-pass filter. On the other hand,for two-photon excitation of the fluorophores, the wavelength of thelight sheets is longer than the wavelength of the fluorescence to bedetected, and thus, the dichroic beam splitter 1823 is set up totransmit the shorter wavelengths to the camera 1969, and the filter 1968can be a short-pass filter or a band-pass filter.

The light source unit 1842 of the second optical arm 1840 includes alight source 1970, an optical control system 1971 that can include anillumination shutter that enables selective activation of the lightsheet 1802B and an illumination filter that blocks light outside of atarget wavelength range, a scanning arrangement 1972 that includes anoptical scanner device (such as one or more rotatable or movablemirrors) 1973 (shown in FIG. 19) and an f-theta lens 1974 (shown in FIG.19). The light sheet 1802B produced by the light source unit 1822 isproduced by scanning an output of the light source 1970. The light beamfrom the light source 1970 is directed to the optical scanner device1973, which deflects the light beam along the x axis to form a lightsheet, which is directed toward the f-theta lens 1974. The opticalscanner device 1973 deflects the incident laser light to produce anangular range that defines the height of the light sheet 1802B along thex axis in the specimen 1801. And, the angle of the light beam that exitsthe optical scanner device 1973 is converted into a displacement alongthe x axis by the f-theta lens 1974 because the optical scanner device1973 is positioned to be located at the focal plane of the f-theta lens1974, thus producing a macroscopic light sheet. Additionally, at the endof the x axis scan, the optical scanner device 1973 steps the light beamfrom the light source 1970 laterally (along a direction perpendicular tothe x axis) so that the light sheet 1802B is displaced along the ydirection to illuminate an adjacent z-x image plane in the biologicalspecimen 1801.

The light source unit 1842 of the second optical arm 1840 shares atleast one component with the detection unit 1844 of the second opticalarm 1840. These shared components are a tube lens 1975 and a sharedobjective 1976. The shared components of the light source unit 1842 andthe detection unit 1844 are separated from the non-shared components ofthe respective light source unit 1842 and the detection unit 1844 by theoptical data separation apparatus, which, in this example, is a dichroicoptical element such as a dichroic beam splitter 1843.

When passing the light sheet 1802B from the non-shared components of thelight source unit 1842 toward the biological specimen 1801, the pair ofthe tube lens 1975 and the shared objective 1976 focuses the macroscopiclight sheet output from the f-theta lens 1974 into the microscopic lightsheet 1802B at the specimen 1801.

The detection unit 1844 of the second optical arm 1840 includes a filter1978, and a camera 1979. Thus, the fluorescence F4 emitted from thespecimen 1801 is collected by the shared objective 1976, and focusedusing the tube lens 1975 onto the camera 1979. The filter 1978 rejectslight of wavelengths outside a wavelength band centered around thewavelength of the fluorescence F4 to be detected. Moreover, the filter1978 can be mounted on a filter wheel with other filters of distinctwavelength ranges so that the wavelength of the rejected light can bechanged by selecting a different filter on the wheel. The filter 1978can be a long pass, a short-pass, or a band-pass filter.

The third optical arm 1830 is arranged opposite the first optical arm1820 so that its optical arm axis is parallel with the first arm axis1821, and the fourth optical arm 1850 is arranged opposite the secondoptical arm 1840 so that its optical arm axis is parallel with thesecond arm axis 1841.

The third optical arm 1830 includes the third light source unit 1832,which includes the following non-shared components: the light source1980, the optical control system 1981, and the scanning arrangement1982, which includes the optical scanner device 1983 and the f-thetalens 1984. The third optical arm 1830 includes the third detection unit1834, which includes the following non-shared components: the filter1988 and the camera 1989. The shared components of the third optical arm1830 are the shared objective 1986 and the tube lens 1985. The sharedcomponents of the light source unit 1832 and the detection unit 1834 areseparated from the non-shared components of the respective light sourceunit 1832 and the detection unit 1834 by the optical data separationapparatus, which, in this example, is a dichroic optical element such asa dichroic beam splitter 1833.

The fourth optical arm 1850 includes the fourth light source unit 1852,which includes the following non-shared components: the light source1990, the optical control system 1991, and the scanning arrangement1992, which includes the optical scanner device 1993 and the f-thetalens 1994. The fourth optical arm 1850 includes the third detection unit1854, which includes the following non-shared components: the filter1998 and the camera 1999. The shared components of the fourth opticalarm 1850 are the shared objective 1996 and the tube lens 1995. Theshared components of the light source unit 1852 and the detection unit1854 are separated from the non-shared components of the respectivelight source unit 1852 and the detection unit 1854 by the optical dataseparation apparatus, which, in this example, is a polychroic (forexample, dichroic) optical element such as a polychroic beam splitter1853.

Each shared objectives 1966, 1976, 1986, 1996 in each respective opticalarm 1820, 1840, 1830, 1850 has a field of view that is at least as greatas the diffraction limit of the optical microscope 1810 and has anumerical aperture that is sufficient to resolve a structure (such as acell) within the biological specimen 1801 that is tracked by thedetection units 1824, 1844, 1834, 1854.

The optical microscope 1810 also includes a chamber 1812 that defines ahollow space and the specimen region 1899 into which the biologicalspecimen 1801 is placed. The biological specimen 1801 is placed on aholder 1866, like the holder 166, which is described above withreference to FIG. 5B.

The optical microscope 1810 also includes a translation system 1870coupled to one or more of the specimen region 1899, and components ofthe optical arms 1820, 1830, 1840, 1850, and configured to translate oneor more of the specimen region 1899, the light sheets 1802A, 1802B,1804A, 1804B produced by the light source units 1822, 1832, 1842, 1852,and the detection units 1824, 1834, 1844, 1854 relative to each otheralong a linear axis without rotating the biological specimen 1801received in the specimen region 1899.

An exemplary translation system 1870 is shown in FIG. 19. Thetranslation system 1870 includes a set of objective translation scanners1914, 1915, 1916, 1917 that are mechanically fixed to respective sharedobjectives 1966, 1976, 1986, 1996. The translation scanners 1914 and1916 are configured to translate (move) the respective shared objectives1966, 1986 in either direction along the y axis (which is the first armaxis 1821), and the translation scanners 1915, 1917 are configured totranslate (move) the respective shared objectives 1976, 1996 in eitherdirection along the z axis. The objective translation scanners can bepiezo-controlled. For example, the objective translation scanners 1914,1915, 1916, 1917 can be P-628.1CD-800 μm travel-range piezo scannersmade by Physik Instrumente (PI) and connected to a LVPZT piezoamplifier/position controller such as the controller E-665-CR made byPI.

The translation system 1870 can also include a set of detectiontranslation scanners 1924, 1925, 1926, 1927 coupled to the components ofthe respective detection units 1824, 1834, 1844, 1854. The translationsystem 1870 can also include a set of light source translation scanners1934, 1935, 1936, 1937 coupled to the components of the respective lightsource units 1822, 1832, 1842, 1852.

The translation system 1870 also includes the optical scanner device(such as a rotatable or movable mirror) 1963, 1973, 1983, 1993 in itsrespective light source unit 1822, 1842, 1832, 1852. The optical scannerdevice 1963, 1973, 1983, 1993 not only scans the light beam from therespective light source 1960, 1970, 1980, 1990 along the x axis to formthe respective light sheet 1802A, 1802B, 1804A, 1804B, but it is alsoconfigured to scan the light sheet 1802A, 1802B, 1804A, 1804B along adirection perpendicular to the x axis. Thus, the optical scanner device1963 scans the light sheet 1802A along the z axis and the opticalscanner device 1983 scans the light sheet 1804A along the z axis duringimaging so that the light sheets step through each of the x-y imageplanes. Moreover, the optical scanner device 1973 scans the light sheet1802B along the y axis and the optical scanner device 1993 scans thelight sheet 1804B along the y axis during imaging so that the lightsheets step through each of the z-x image planes.

Referring to FIG. 20, the electronics controller 1880 can include areal-time controller 2005 such as the real-time controller 650 describedabove. The real-time controller 2005 would include additional componentsfor the new optical and mechanical components within each of the opticalarms 1820, 1830, 1840, 1850. The electronics controller 1880communicates with the computational system 1890 to coordinate thesimultaneous image acquisition workflow. The electronics controller 1880can also include dedicated camera links 2010 for connecting directly tothe cameras 1969, 1979, 1989, 1999 within each optical arm 1820, 1830,1840, 1850.

All of the optical, mechanical, and electrical components within theoptical microscope 1810 are connected to components within thecontroller 2005 to provide real-time control in each of the “arms” ofthe microscope 1810. The controller 2005 also includes an automatedalignment module for rapid relative positioning and orientation of thesets of light sheets 1802A, 1804A and 1802B, 1804B and the respectivefocal planes. The controller 2005 modifies the various optical,mechanical, and electrical components of the optical microscope toprovide twenty degrees of freedom of motion to optimize or increase thefluorescence signal F2, F3, F4, F5 detected by each detection unit 1824,1834, 1844, 1854.

In one general aspect, the real-time electronics controller 2005 is areal-time controller such as a PXI-8110 2.2 GHz Quad Core embeddedcontroller by National Instruments Corporation of Austin, Tex. Thiscontroller can run a LabVIEW Real-Time operating system, and is equippedwith three I/O interface boards (such as the PXI-6733 high-speed analogoutput 8-channel board, also by National Instruments) linked to BNCconnector blocks (such as the BNC-2110 shielded connector block, also byNational Instruments) as well as a serial interface board (such asPXI-8432/2, also by National Instruments). The real-time controller 2005communicates with the computational system 1890 by way of a high-speeddata transmission such as Gigabit Ethernet.

In addition to the real-time controller 2005, the electronics controller1880 can also include a motion controller and analog and digitalinput/output channels on separate controllers. The motion controller canbe a PXI-7354 motion controller from National Instruments and it caninclude a plurality of I/O controllers such as the PXI-6733 controllersfrom National Instruments, having eight analog outputs and eight digitalI/O channels each.

All time-critical tasks can be performed within the electronicscontroller 1880, with the remaining tasks (such as collecting andvisualizing frames recorded by the cameras) being performed by thecomputational system 1890.

The computational system 1890 can include a computer such as aworkstation that has the ability to store, retrieve, and process data.Thus, the computer includes hardware such as one or more output devices2015 such as a monitor or a printer; one or more user input interfaces2020 such as a keyboard, a mouse, a touch display, or a microphone; oneor more processing units 2025, including specialized workstations forperforming specific tasks; memory (such as, for example, random-accessmemory or read-only memory or virtual memory) 2030; and one or morestorage devices 2035 such as hard disk drives, solid state drives, oroptical disks. The processing units can be stand-alone processors, orcan be sub-computers such as workstations in their own right.

The specialized workstations include an on board processing unit, inaddition to memory, and software for running specific tasks. Thespecialized workstations include an image acquisition workstation 2040,an image processing workstation 2045, and an optional image segmentationand tracking workstation 2050. Additionally, the specializedworkstations include a multiview image alignment workstation 2055, whichperforms the task of combining the data from each of the views that areobtained. The combining of the data may include determining which viewprovides the best or clearest image of the structure in the biologicalspecimen 1801 that is being tracked. The multiview image alignmentworkstation 2055 may run a multiview deconvolution algorithm on the datasets from each of the views to perform this alignment and combination.

The storage devices 2035 include, among others, an image data managementstorage unit 2060 that receives information from the image acquisitionworkstation 2040 and also is equipped to receive its own processors, foradditional processing capabilities.

The workstations include their own software modules, which each includea set of instructions that, when executed, cause the hardware within theelectronics controller 1880 to perform various actions, as discussedbelow.

The computational system 1890 is designed for high-speed imagingexperiments with up to several days of continuous image acquisition. Thecomputational system 1890 can be set up to record more than one millionhigh-resolution images in uninterrupted high-speed imaging sessions witha total data set size of, for example, ten terabytes (TB) per specimenif each recorded image is about 10 megabytes (MB) in size and thestorage capacity is 10 TB. To permit high-speed recording, each of thelines connecting the image acquisition workstation 2040 to the imagedata management unit 2060 and to the controller 2005 can be set up as aglass fiber network pipeline, which can provide 10 gigabit/second dataspeed, thus allowing recording up to ten million high-resolution images(for example, each image being 10 MB in size) or 100 terabytes or moreper specimen 1801 for long-term imaging sessions. A maximum recordingcapacity of one petabyte can be realized if the image data managementunit 2060 uses a three-dimensional wavelet compression technique havingan average ratio of 10:1.

The image acquisition workstation 2040 and the multiview imageprocessing workstation 2045 are developed for content-based imageregistration and multiview image fusion, respectively, which efficientlyincorporates prior knowledge of the optical implementation to processraw image data at a rate of about 200 megabyte/seconds. The imageacquisition workstation 2040 is capable of real-time image registrationand integrates with the image processing workstation 2045 forlarge-scale data management. Since the computational system 1890acquires multiple views simultaneously, fast and accurate imageregistration (alignment of images) within the image acquisitionworkstation 2040 is achieved without the need of fiducial markers in theimaging volume IV.

In one implementation, the computational system 1890 is a Windows-basedpersonal computer having twelve physical core processors within theprocessing units 604 operating at 3.3 GHz and accessing 64 gigabytes(GB) of RAM in memory 606. The computational system 1890 can run anysuitable operating system, such as Windows 7 (64-bit), and can run anysuitable applications such as LabVIEW development suite (64-bit).

In the following, we provide a short description of the workflow in thecomputational system 1890. In general, the image acquisition workstation2040 performs a registration of images, which is a process that alignsthe images from the cameras in the opposing optical arms.

Thus, the images from cameras 1969 and 1989 are aligned and the imagesfrom cameras 1979 and 1999 are aligned. In general, the image processingworkstation 2045 performs a fusion of the aligned images, which is aprocess of combining the registered images into a single representationor image. In particular, images are fused by combining the informationcontent of the images into a single image. Details about fusion arediscussed below.

The image data management unit 2060 provides a high-throughput imagestorage pipeline for sustained data streaming at 600 megabytes/second;uninterrupted long-term image acquisition of 100 terabyte sized datasets, and a wavelet-based lossless image compression (for example,ten-fold).

The image acquisition workstation 2040 relays the raw multiview datastream to the image processing workstation 2045 and the image datamanagement unit 2060 by way of optical fibers. The software used in theworkstations can be written in, for example, Matlab and C++ forproviding high-throughput multiview image processing and real-time imagedata management.

The light sources 1960, 1970, 1980, 1990 can all be from the same masterlight source 1898, which can be a laser system. For example, the masterlight source 1898 can include two Omicron SOLE-6 engines (4 wavelengthsin each engine), with a communication interface to microscopeinput/output hub: one cable for digital signal transmission (on/off) andone cable for analog signal transmission (such as laser power) perwavelength and SOLE; one USB cable per SOLE for initial, manual laserconfiguration. The output from the master light source 1898 is splitinto four beams, each constituting the respective light source 1960,1970, 1980, 1990. In other implementations, a dedicated laser system (orlight source) can be provided for each light source 1960, 1970, 1980,1990, and each dedicated laser system could be individually controllableby the electronics controller 1880 and the computational system 1890.

The output from each light source 1960, 1970, 1980, 1990 can becontrolled using the optical shutter 1961, 1971, 1981, 1991, whichcontrols the timing of the respective light sheet 1802A, 1802B, 1804A,1804B relative to the other light sheets that reach the biologicalspecimen 1801. This optical shutter 1961, 1971, 1981, 1991 is able toblock light that is directed through an aperture, and the blocking ofthe light can be synchronized, asychronized, or controlled depending onthe application.

In combination with the optical shutters in the other optical arms, theoptical shutters within the respective light source units 1822, 1832,1842, 1852 control which of the light sheets (for example, only thelight sheet 1802A, only the light sheet 1802B, or both the light sheet1802A and the light sheet 1802B) illuminate the specimen 1801 at any onemoment. The illumination at the specimen 1801 can therefore be performedsynchronously or asynchronously from either side. A driver or actuatorconnected to the electronics controller 1880 operates each laser shutter1961, 1971, 1981, 1991 under control from the electronics controller1880.

The polychroic beam splitter 1823, 1833, 1843, 1853 can be, for example,polychroic beam splitters models ZT488/561rpc or ZT488/594rpc fromChroma Technology Corp (of Bellows Falls, Vt.). The filters 1968, 1978,1988, 1998 can be, for example, BrightLine fluorescence filters fromSemrock, Inc., a part of IDEX Corporation of Lake Forest, Ill. Each ofthe shared objectives 1966, 1976, 1986, 1996 can be a microscopeobjective that includes a single lens, or a combinations of lenses andother optical elements.

The cameras 1969, 1979, 1989, 1999 can be scientific CMOS (complementarymetal oxide semiconductor) image sensors (cCMOS sensors). For example,the cameras 1969, 1979, 1989, 1999 can be Hamamatsu Orca Flash 4.0 v2sCMOS cameras connected to frame grabbers in the computational system1890 by way of CameraLink cables.

The illumination optical shutters 1961, 1971, 1981, 1991 can be UniblitzLS6ZM2-100 laser shutters controlled by two Uniblitz VMM-D3three-channel shutter drivers (two shutters per driver, third channel isempty); a communication interface to microscope I/O hub (the electronicscontroller 1880); with one flying-leads cable per shutter, transmittingdigital signals (5V TTL).

The filter wheels that can be used for the filters 1968, 1978, 1988,1998 in the detection units and also for the illumination filters withinthe optical control system of the light source units can be Ludl filterwheels (four for illumination and four for detection). These can becontrolled by four dual-channel MAC6000 DC controllers (one forillumination and one for detection filter wheel per MAC6000); and acommunication interface to microscope I/O hub (the electronicscontroller 1880); and one RS-232 serial cable per filter wheel.

The scanning arrangement 1962, 1972, 1982, 1992 can be driven usinggalvanometer drivers, with one Cambridge 6220 XY scanner connected to aMicroMax series 673xx dual axis servo driver in each optical arm; and acommunication interface to microscope I/O hub (the electronicscontroller 1880); and two cables per scanner, transmitting analogsignals (0-10V).

The sample 1801 can be positioned within the chamber 1812 using a samplepositioning system that includes a four-axis motion controller, PIC-884, connected to three PI M-111.2DG translation stages and one PIM-116 rotation stage; and communication interface to microscope I/O hub(the electronics controller 1880).

In summary, the microscope system 1700, 1800 is set up such that eachoptical arm is now equipped with both a light source unit and adetection unit. The microscope system 1700, 1800 records thefluorescence emitted from the biological specimen 1701, 1801 over itsentire volume with one illumination/detection configuration bytranslating the light sheets and objectives as discussed in greaterdetail below (relative to the biological specimen) along the detectionaxes that are parallel with the second optical axis 1741, 1841. Forexample, with reference to FIG. 17, the light source units 1722, 1732can produce and direct the respective light sheets 1702A, 1704A to thebiological specimen 1701 and the detection units 1744, 1754 detect theemitted respective fluorescence F4, F5 from the biological specimen1701. The microscope system 1700 is set up to switch off the active pairof light source units (units 1722, 1732) and to activate the other pairlight source units 1742, 1752 within the other optical arms 1740, 1750;and to switch off the active pair of detection units (units 1744, 1754)and to activate the other pair of detection units 1724, 1734. Then,microscope system 1700 records the fluorescence emitted from thebiological specimen over its entire volume a second time, by translatingthe light sheets and objectives as discussed below (relative to thebiological specimen) along the detection axes that are parallel with thefirst optical axis 1721 (and also perpendicular to thepreviously-scanned detection axis that is parallel with the secondoptical axis 1741).

Referring to FIG. 22, a procedure 2200 is performed by the microscopesystem 1800 to image the complex biological specimen 1801. Initially,the biological specimen 1801 is prepared (2202). The biological specimen101 is prepared (2202) by chemically and biologically preparing thespecimen, physically transferring or mounting the specimen to the holder1866, and placing the holder 1866 inside the chamber 1812.

Next, the optical microscope 1810 is prepared (2204). For example, theoptical microscope can be prepared (2204) by adjusting properties (suchas the alignment) of the light sheets 1802A, 1802B, 1804A, 1804B.

Once the optical microscope 1810 is prepared (2204), the one or morefirst light sheets 1802A, 1804A are generated (2206). The one or morefirst light sheets 1802A, 1804A are directed toward the biologicalspecimen 1801 such that there is spatial and temporal overlap within thespecimen 1801 (2208). The temporal overlap of the one or more firstlight sheets 1802A, 1804A is less than a resolution time thatcorresponds to a spatial resolution limit of the optical microscope1810. And, the spatial displacement in a tracked structure of thebiological specimen 1801 during this time shift is less than the spatialresolution limit of the optical microscope 1810.

The one or more first light sheets 1802A, 1804A are directed alongrespective paths, with both paths being parallel with the firstillumination axis, which is parallel with the first arm axis 1821. Theone or more first light sheets 1802A, 1804A optically interact with atleast a portion of the biological specimen 1801 in a first image plane,which is in the x-y plane. The one or more first light sheets 1802A,1804A can have different polarization states from each other.

The beginning of the illumination and recording of the fluorescence canstart at the moment the fertilized egg is formed, to enable imaging ofthe biological specimen 1801 in its development from a fertilized egg toa complex system.

Images of the fluorescence F4, F5 that is emitted from the biologicalspecimen 1801 are recorded at each of a plurality of first views by thecameras 1979, 1999, respectively, of respective optical arms 1840, 1850(2210) along the detection axes of the cameras 1979, 1999. This includescapturing the fluorescence F4 with the shared objective 1976 anddirecting this captured fluorescence F4 from the shared objective 1976to the camera 1979 that records the images of the fluorescence F4. Thisalso includes capturing the fluorescence F5 with the shared objective1996 and directing this captured fluorescence F5 from the sharedobjective 1996 to the camera 1999 that records the images of thefluorescence F5. Images of the fluorescence F4, F5 can be recorded ateach of the plurality of first views by the cameras 1979, 1999 byensuring that the shared objectives 1976, 1996 of the respective opticalarms 1840, 1850 are aligned so that their focal planes overlap with theplane of the light sheets 1802A, 1804A.

Steps 2206-2210 are performed at each z position as the one or moregenerated first light sheets 1802A, 1804A and the shared objectives1976, 1996 of the optical arms 1840, 1850 are translated in steps alongthe z axis so as to image the entire biological specimen 1801. Theprocedure 2200 may also include repositioning the biological specimen1801 after steps 2206-2210 are performed in the y and z directions.

Next, one or more second light sheets 1802B, 1804B are generated (2212).In some implementations described below, the one or more second lightssheets 1802B, 1804B are generated (2212) after the generation of the oneor more first light sheets 1802A, 1804A (2206) and after thede-activation of the one or more respective light source units 1822,1832. In other implementations that are described below, the one or moresecond light sheets 1802B, 1804B are generated (2212) during thegeneration of the one or more first light sheets 1802A, 1804A (2206).The one or more second light sheets 1802B, 1804B can have differentpolarization states from each other.

The one or more second light sheets 1802B, 1804B are directed toward thebiological specimen 1801 such that there is spatial and temporal overlapwithin the specimen 1801 (2214). The temporal overlap of the one or moresecond light sheets 1802B, 1804B is within a time shift that is lessthan a resolution time that corresponds to a spatial resolution limit ofthe optical microscope 1810. And, the spatial displacement in a trackedstructure of the biological specimen 1801 during this time shift is lessthan the spatial resolution limit of the optical microscope 1810.

The one or more second light sheets 1802B, 1804B are directed alongrespective paths that are parallel with a second illumination axis thatis parallel with the second arm axis 1841 such that the one or moresecond light sheets 1802B, 1804B optically interact with at least aportion of the biological specimen in a second image plane that isparallel with the z-x plane. The second illumination axis is notparallel with the first illumination axis. Images of the fluorescenceF2, F3 that is emitted from the biological specimen 1801 are recorded ateach of a plurality of second views by the cameras 1969, 1989,respectively, of respective optical arms 1820, 1830 (2216) along thedetection axes of the cameras 1969, 1989. Images of the fluorescence F2,F3 can be recorded at each of the plurality of second views by thecameras 1969, 1989 by ensuring that the shared objectives 1966, 1986 ofthe respective optical arms 1820, 1830 are aligned so that their focalplanes overlap with the plane of the light sheets 1802B, 1804B.

Steps 2212-2216 are performed at each y position while the one or moregenerated second light sheets 1802B, 1804B and the shared objectives1966, 1986 of the respective optical arms 1820, 1830 are translatedalong the y axis in steps so as to image the entire biological specimen1801. The procedure 2200 may also include repositioning the biologicalspecimen 1801 after steps 2212-2216 are performed in the y and zdirections.

Imaging (which includes steps 2206-2216) of the biological specimen 1801continues until a pre-determined time of development of the biologicalspecimen. For example, imaging can continue until the onset of strongmuscle contractions in the developing embryo (biological specimen); atthat point, imaging can be stopped because the specimen 1801 becomesmore physically active and can be more difficult to image. However, itis possible that imaging could continue past this developmental point.

During the procedure 2200, it may be useful to reset the relativealignment between the one or more first light sheets 1802A, 1804A andthe biological specimen 1801, and the relative alignment between the oneor more second light sheets 1802B, 1804B and the biological specimen1801 before the fluorescence is once again recorded. Once the imaging ofthe biological specimen 1801 is completed (namely, all of thefluorescence images are recorded 2210 and 2216), the image of thebiological specimen 1801 is created (2218). The image of the biologicalspecimen 1801 contains a set of tracked structures (such as cells)within the biological specimen 1801 that develop over the time duringwhich the biological specimen 1801 is scanned. Thus, for example, atracked structure may move from one part of the biological specimen 1801to another part of the biological specimen 1801 and the path can bevisualized during the imaging. Or, a tracked structure, such as a cell,can divide by mitosis into two cells, and each of those two cells can betracked.

It takes some amount of time to switch from imaging using steps2206-2210 (which involves translating the light sheets 1802A, 1804A andthe objectives 1976, 1996 along the z axis) and imaging using steps2212-2216 (which involved translating the light sheets 1802B, 1804B andthe objectives 1966, 1986 along the y axis). It preferably should takeno more than a few milliseconds to switch from each of these imagingschemes. And scan of the entire biological specimen 1801 (performed ateach x-y plane and at each z-x plane) can take a few milliseconds oreven more, depending on the volume of the biological specimen 1801 thatis being imaged and the density of the image planes (which is thedistance between each image plane that is scanned).

There are different modes of operation of the microscope system 1800,and these different modes are different ways to scan and image thebiological specimen 1801 using the procedure 2200. The mode of operationis governed by how the information that is recorded (the fluorescence)is separated out by the microscope system 1800 so that an accurateanalysis of the information can proceed. The computational system 1890needs to be able to discriminate between how the fluorescence isproduced in order to analyze the information accurately.

In some implementations, the recorded information can be separated outin color or spectrum space, which means that the information can be atdistinct wavelengths. For example, if the one or more first light sheets1802A, 1804A are at a different wavelength from the one or more secondlight sheets 1802B, 1804B, then the one or more first light sheets1802A, 1804A can be configured to excite a first set of fluorophoreswithin the biological specimen 1801 that emit fluorescence F4, F5 at afirst wavelength and the one or more second light sheets 1802B, 1804Bcan be configured to excite a second set of fluorophores within thebiological specimen 1801 that emit fluorescence F2, F3 at a secondwavelength. Because the first wavelength is distinct from the secondwavelength, the computational system 1890 can distinguish between whichlight sheets produce which fluorescence during the analysis of theinformation.

In other implementations, the recorded information can be separated outin temporal space, which means that the information can be obtainedsequentially. It is not necessary to operate a distinct wavelengths ifthe information is separated out in temporal space. In still otherimplementations, the recorded information can be separated out inspatial space, which means that the information is separated using aconfocal aperture. It is not necessary to operate a distinct wavelengthsif the information is separated out in spatial space.

Next, these exemplary modes of operation to scan and image thebiological specimen 1801 are described.

Referring to FIGS. 23A-26, in a first exemplary mode of operation theinformation is separated out in temporal space because the one or morefirst light sheets 1802A, 1804A and the one or more second light sheets1802A, 1804A have the same wavelength and they are operatedsequentially.

With specific reference to FIGS. 23A-C and 24, the light source unit1822 of the first optical arm 1820 is activated to produce one of thefirst light sheets 1802A, and the light source unit 1832 of the thirdoptical arm 1830 is activated to produce the other of the first lightsheets 1804A, which are directed to the biological specimen 1801 inopposite directions along the paths that are parallel with the firstillumination axis (and the first arm axis 1821 and the y axis). The oneor more first light sheets 1802A, 1804A spatially and temporally overlapwithin the specimen 1801, as shown schematically in FIGS. 23A-C.Initially, the one or more first light sheets 1802A, 1804A can bedirected along a base x-y image plane as shown in FIG. 23A. Thefluorescence F4, F5 emitted from the biological specimen 1801 isdetected by the respective cameras 1979, 1999 as images, and this imagedata is received by and properly processed by the electronics controller1880, and then recorded within the computational system 1890.

The fluorescence at each x-y image plane of the specimen 1801 isrecorded until the entire biological specimen 1801 is captured. Forexample, after the fluorescence F4, F5 at the image plane shown in FIG.23A is recorded, the relative placement along the z axis between thebiological specimen 1801 and the one or more first light sheets 1802A,1804A is modified by a step along the z axis so that the next x-y imageplane can be recorded in a z axis translation scheme. In this way, thefluorescence emitted from the biological specimen 1801 is recordedincrementally at each of the x-y image planes of the biological specimen1801. For example, in FIG. 23B, the fluorescence F4, F5 is recorded atthe x-y image plane located near a central region of the biologicalspecimen 1801; and in FIG. 23C, the fluorescence F4, F5 emitted from thebiological specimen 1801 is recorded at the x-y image plane located at atop region of the biological specimen 1801.

During this time while the one or more first light sheets 1802A, 1804Aare interacting with the biological specimen 1801 and the fluorescenceF4, F5 is being recorded, the light source unit 1842 within the secondoptical arm 1840 is de-activated and the light source unit 1852 withinthe fourth optical arm 1850 is de-activated so that the one or moresecond light sheets 1802B, 1804B are not being generated.

Once the fluorescence F4, F5 emitted from the biological specimen 1801is recorded at each x-y image plane, the light source unit 1822 withinthe first optical arm 1820 is de-activated and the light source unit1832 within the third optical arm 1830 is de-activated. Next, as shownin FIGS. 25A-C and 26, the light source unit 1842 of the second opticalarm 1840 is activated to produce one of the second light sheets 1802B,and the light source unit 1852 of the fourth optical arm 1850 isactivated to produce the other of the second light sheets 1804B, both ofwhich are directed to the biological specimen 1801 in oppositedirections along the paths that are parallel with the secondillumination axis (and the second arm axis 1841 and the z axis). The oneor more second light sheets 1802B, 1804B spatially and temporallyoverlap within the specimen 1801, as shown schematically in FIGS. 25A-C.Initially, the one or more second light sheets 1802B, 1804B can bedirected along a base z-x image plane as shown in FIG. 25A. Thefluorescence F2, F3 emitted from the biological specimen 1801 isdetected by the respective cameras 1969, 1989 as images, and this imagedata is received by and properly processed by the electronics controller1880, and then recorded within the computational system 1890.

The fluorescence F2, F3 at each z-x image plane of the specimen 1801 isrecorded until the entire biological specimen 1801 is captured. Forexample, after the fluorescence F2, F3 at the image plane shown in FIG.25A is recorded, the relative placement along the y axis between thebiological specimen 1801 and the one or more second light sheets 1802B,1804B is modified by a step along the y axis so that the next z-x imageplane can be recorded in a y axis translation scheme. In this way, thefluorescence F2, F3 emitted from the biological specimen 1801 isrecorded incrementally at each of the z-x image planes of the biologicalspecimen 1801. For example, in FIG. 25B, the fluorescence F2, F3 isrecorded at the z-x image plane located near a central region of thebiological specimen 1801; and in FIG. 25C, the fluorescence F2, F3emitted from the biological specimen 1801 is recorded at the z-x imageplane located at a top region of the biological specimen 1801.

In this first exemplary mode of operation, scanning and recording canalternate between activation of the light source units 1842, 1852 withinthe respective optical arms 1840, 1850 and recording using the detectionunits 1824, 1834 within the respective optical arms 1820, 1830 (duringwhich time the light source units 1822, 1832 are de-activated), andactivation of the light source units 1822, 1832 within the respectiveoptical arms 1820, 1830 and recording using the detection units 1844,1854 within the respective optical arms 1840, 1850 (during which timethe light source units 1842, 1852 are de-activated). Moreover, these twoperpendicular illumination and detection schemes can be alternated aftereach complete scan through the biological specimen 1801.

It is alternatively possible, in this first exemplary mode of operation,to interweave the z axis translation scheme and the y axis translationscheme by alternating between the two perpendicular illumination anddetection schemes after each step in the respective direction (eithereach y step or each z step). For example, after one z step, instead ofgoing to the next z step, the scanning and recording switches arms, andthe z step becomes inactive and y step is recorded. After that, thescanning and recording switches arms again, and the next z step isrecorded (while the y step is inactive), until all steps within thebiological specimen 1801 are scanned and recorded.

Thus, for example, the following exemplary sequence can transpire inthis interweaving scheme:

I. a scan and recording can occur at an x-y image plane IP(x-y)1 inwhich the one or more first light sheets 1802A, 1804A are directed tothe biological specimen 1801 and the cameras 1979, 1999 recordrespective fluorescence F4, F5;

II. a scan and recording can occur at a z-x image plane IP(z-x)1 inwhich the one or more second light sheets 1802B, 1804B are directed tothe biological specimen 1801 and the cameras 1969, 1989 recordrespective fluorescence F2, F3;

III. a scan and recording can occur at an x-y image plane IP(x-y)2 inwhich the one or more first light sheets 1802A, 1804A are directed tothe biological specimen 1801 and the cameras 1979, 1999 recordrespective fluorescence F4, F5; and

IV. a scan and recording can occur at a z-x image plane IP(z-x)2 inwhich the one or more second light sheets 1802B, 1804B are directed tothe biological specimen 1801 and the cameras 1969, 1989 recordrespective fluorescence F2, F3.

During the scan and recording at the x-y image planes IP(x-y)1 andIP(x-y)2, the light source units 1822, 1832 are activated and the lightsource units 1842, 1852 are de-activated; and during the scan andrecording at the z-x image planes IP(z-x)1 and IP(z-x)2, the lightsource units 1842, 1852 are activated and the light source units 1822,1832 are de-activated.

Referring to FIG. 27, in another mode of operation, the recordedinformation is separated out in color or spectrum space, which meansthat the information is at distinct wavelengths. In this mode, the oneor more first light sheets 1802A, 1804A are at a first illuminationwavelength λI1, and the one or more second light sheets 1802B, 1804B areat a second illumination wavelength λI2. In this way, the one or morefirst light sheets 1802A, 1804A are configured to excite a first set offluorophores within the biological specimen 1801 that only emitfluorescence F4, F5 when excited with the wavelength λI1, and the one ormore second light sheets 1802B, 1804B are configured to excite a secondset of fluorophores within the biological specimen 1801 that only emitfluorescence F2, F3 when excited with the wavelength λI2. Moreover, thefirst set of fluorophores that are excited by the light of the firstillumination wavelength λI1 emit fluorescence F4, F5 at a firstdetection wavelength λD1, and the second set of fluorophores that areexcited by the light of the second illumination wavelength λI2 emitfluorescence F2, F3 at a second detection wavelength λD2. Because thefirst illumination wavelength λI1 is distinct from the secondillumination wavelength λI2, and the first detection wavelength λD1 isdistinct from the second detection wavelength λD2, the computationalsystem 1890 is able to distinguish between which light sheets producewhich fluorescence during the analysis of the recorded images.

This mode of operation can be useful when different types of nearbyinformation need to be extracted from the same biological specimen 1801in the same experiment. Because resolution of the optical microscope1810 is at best diffraction limited, the fluorophores of the two typesof information may be closer together, which can make it difficult toeasily interpret the resulting image data without some separation of theinformation. For example, it is possible that morphological informationabout cell nuclei needs to be imaged and morphological information aboutthe cells' outer plasma membranes needs to be imaged at the same timeand these two cellular structures are so close to each other that itbecomes difficult to separate the information from the two structureswithout the use of different spectral domains. Thus, these two cellulardomains can be labeled in different spectral domains (wavelengths) inorder to obtain useful image data that enables the analysis of thesenearby structures independently. Another example of two types ofcellular domains are labels for microtubule cytoskeletons and the actincytoskeleton. Any structures that are close together are easier toresolve when labeled in different spectral domains (wavelengths). On theother hand, for structures that are farther apart from each other, suchas the nuclei of all of an embryo's cells, one can use a single colorlabel (one wavelength) throughout the biological sample 1801 and performimaging using the other modes of operation. Structures can be fartherapart from each other if they are separated from each other by adistance that is at least as great as the axial resolution of theoptical microscope 1810 in order to be identifiable as separate objectswhen using the same color channel to label both structures. In practice,this axial resolution can be a few micrometers.

Referring to FIG. 27, because the information is separated in spectrumspace, it is possible to scan the one or more first light sheets and theone or more second light sheets simultaneously, and to record thefluorescence F2, F3, F4, F5 simultaneously. As time progresses, the oneor more first light sheets 1802A, 1804A and the shared objectives 1976,1996 of the optical arms 1840, 1850 are translated in steps along the zaxis in a z axis translation scheme; and the one or more second lightsheets 1802B, 1804B and the shared objectives 1966, 1986 of therespective optical arms 1820, 1830 are translated in steps along the yaxis in a y axis translation scheme. Thus, the z axis translation schemeand the y axis translation scheme occur at the same time. Moreover, inthis mode of operation, both the one or more first light sheets 1802A,1804A and the one or more second light sheets 1802B, 1804B are directedto the biological specimen 1801 at the same time; therefore it ispossible that both the first and second sets of fluorophores are excitedsimultaneously during scanning and recording.

Thus, all optical arms 1820, 1830, 1840, 1850 carry out both theillumination and detection operations simultaneously. One design issuewith this mode of operation is that as the shared objectives 1976, 1996of the optical arms 1840, 1850 are translated, the focal spots of therespective light sheets 1802B, 1804B are shifted accordingly, and as theshared objectives 1966, 1986 of the optical arms 1820, 1830 aretranslated, the focal spots of the respective light sheets 1802A, 1804Aare shifted accordingly. Though not required for operation of themicroscope system 1800, this focal spot shift can be compensated forbecause the detected fluorescence F2, F3, F4, F5 may not be optimallyimaged at the respective cameras 1969, 1979, 1989, 1999, which arealways detecting fluorescence because all illumination and detectionoperations are carried out simultaneously, and it is possible that suchfocal spot shift can reduce the axial resolution when compared with animaging performed that lacks such focal spot shift. For example, thereduction in axial resolution can arise because the fluorescence isbeing created from the fluorophores external to the thinnest portion ofeach light sheet because the light sheets are not symmetricallypositioned around the center of the field-of-view but are shifted towardone side or the other depending on the progression of the volume scan.

Thus, the microscope system 1800 can include an additional compensatoryoptical module in each light source unit 1822, 1832, 1842, 1852.Referring to FIG. 29, an expanded view of the optical arm 1820 includesthe compensatory optical module 2900, which is placed in the pathbetween the dichroic beam splitter 1823 and the scanning arrangement1962, in a plane that is conjugate to the back focal plane of sharedobjective 1966. The compensatory optical module 2900 can include, forexample, an electronic tunable lens that is flanked by lenses. Theelectronic tunable lens can be a fast electrically tunable lens modelnumber EL-10-30 made by Optotune AG of Dietikon, Switzerland. Thecompensatory optical module 2900 translates the focal spot of the sharedobjective (such as the shared objective 1966 shown in FIG. 29) by anequal and opposite extent to the amount produced by the translation ofthe shared objective 1966.

Referring to FIG. 30, in another mode of operation, the recordedinformation is separated out in spatial space, which means that theinformation is separated out in each optical arm 1820, 1830, 1840, 1850using a respective confocal aperture 3025, 3035, 3045, 3055, and the oneor more second light sheets 1802B, 1804B are staggered in space from theone or more first light sheets 1802A, 1804A so that they do not overlapwhen they pass through the biological specimen 1801 along theirnon-parallel illumination axes. In this mode of operation, both the oneor more first light sheets 1802A, 1804A and the one or more second lightsheets 1802B, 1804B can be at the same illumination wavelength λI, andthe fluorophores can release light of the same detection wavelength λD.Moreover, all of the optical arms 1820, 1830, 1840, 1850 can carry outboth the illumination and detection operations simultaneously in thismode of operation, as described above with respect to FIG. 28.

Referring to FIG. 29, which shows an exploded view of the optical arm1820, the confocal aperture 3025 is placed in a plane that is conjugateto the sample plane. The confocal aperture 3025 can be built in to thecamera 1969, and it can be set up to translate along the z axis to tracea path that follows the location of the fluorescence F2. Referring toFIG. 31A, the confocal aperture 3025 defines an opening 3110 throughwhich light passes into the sensor 3115 of the camera 1969. The opening3110 moves along with the path of the fluorescence F2 that is supposedto be detected at the sensor 3115 and is wide enough to pass thefluorescence F2, which is focused to the location of the opening 3110,as shown in FIGS. 31B and 31C. In particular, the fluorescence F2 is dueat least in part to the interaction of the sample 1801 with the lightsheet 1802B, and any light (such as fluorescence F2) originating fromthe location of the focus of the light sheet 1802B in the sample 1801 ispassed through the opening 3110 of the aperture 3025 to the sensor 3115,except that the light from the light sheet 1802B itself is blocked bythe filter 1968, which is not shown in FIGS. 31A-C. Thus, anyfluorescence such as F4 that is emitted from fluorophores that interactwith the light sheet 1802A are blocked by the aperture 3025 and do notpass through the opening 3110 because the light sheet 1802A does notoverlap with or cross the light sheet 1802B within the biologicalspecimen 1801.

Referring to FIG. 32, in order to use the confocal aperture 3025 toprovide a spatial way to separate information, the one or more secondlight sheets 1802B, 1804B are staggered in space from the one or morefirst light sheets 1802A, 1804A so that they do not overlap when theypass through the biological specimen 1801 along the non-parallelillumination axes. FIG. 32 shows how the light beams from the lightsources in each of the optical arms are formed and also staggered usingthe scanning arrangements. The four scanning arrangements 1962, 1982,1972, 1992 of the respective optical arms 1820, 1830, 1840, 1850 areshown in FIG. 32 as being directed along the same illumination axis forillustration purposes only to convey how the respective light sheetsthat are produced are staggered. The direction of scanning in thisexample is along the x axis, however the four scanning arrangements1962, 1982, 1972, 1992 are arranged along different illumination axes.

In this example, the beam from the light source 1960 of the light sourceunit 1822 of the first optical arm 1820 is deflected along (and scannedacross) the x axis by the optical scanner device (such as the rotatableor movable mirror) 1963 toward the f-theta lens 1964. Additionally, thebeam from the light source 1980 of the light source unit 1832 of thethird optical arm 1830 is deflected along (and scanned across) the xaxis by the optical scanner device (the movable mirror) 1983 toward thef-theta lens 1984. The angle of the light beam that exits the opticalscanner device 1963, 1983 is converted into a displacement along the xaxis by the respective f-theta lens 1964, 1984 because the opticalscanner device 1963, 1983 is positioned to be located at the focal planeof the f-theta lens 1964, 1984 thus producing a macroscopic light sheet.As shown, the macroscopic light sheets produced in the optical arms1820, 1830 are overlapping and not displaced from each other. On theother hand, the macroscopic light sheets produced in the optical arms1840, 1850 are displaced from the macroscopic light sheets produced inthe optical arms 1820, 1830 by a displacement D. Thus, the beam of thelight sheet 1802B is offset from the beam of the light sheet 1802A by adisplacement D′ inside the biological specimen 1801, as shown in FIGS.31B and C. Additionally, the beam of the light sheet 1804B is alsooffset from the beam of the light sheet 1802A and the beam of the lightsheet 1804A and the beam of the light sheet 1802B is also offset fromthe beam of the light sheet 1804A, even though those are not shown inFIGS. 31B and C.

Referring to FIG. 33, the image of the biological specimen 1801 iscreated using a procedure 2218. The procedure 2218 includes forming afirst image that is produced from the interaction of the one or morefirst light sheets 1802A, 1804A with the biological specimen 1801(3305). Forming the first image involves an analysis of the informationthat is recorded at the cameras 1979, 1999 of the respective detectionunits 1844, 1854. This analysis proceeds in the manner described usingthe procedure 716, and reference is made to the description associatedwith FIG. 9, and thus will not be duplicated here. The computationalsystem 1890 fuses aligned (and registered) fused data sets into thefirst image, which is representative of the image x-y plane of thebiological specimen 1801 that corresponds to the spatial and temporaloverlap of the light sheets 1802A, 1804A within the specimen 1801.

Next, the procedure 2218 includes forming a second image that isproduced from the interaction of the one or more second light sheets1802B, 1804B with the biological specimen 1801 (3310). Forming thesecond image involves an analysis of the information that is recorded atthe cameras 1969, 1989 of the respective detection units 1824, 1844.This analysis proceeds in the manner described using the procedure 716,and reference is made to the description associated with FIG. 9, andthus will not be duplicated here. The computational system 1890 fusesaligned (and registered) fused data sets into the second image, which isrepresentative of the image z-x plane of the biological specimen 1801that corresponds to the spatial and temporal overlap of the light sheets1802B, 1804B within the specimen 1801.

Steps 3305 and 3310 are performed for each x-y plane that is recordedand each z-x plane that is recorded until the entire data set isanalyzed and a set of first and second images are created. Generally,the tracked structure in the imaged sample z-volume (obtained from step3305) is not going to be perfectly identical to the tracked structure inthe imaged sample y-volume (obtained from step 3310), because thebiological specimen 1801 is alive and changes and moves as a function oftime and the imaging in the two different views may not be performedsimultaneously. In order to combine all views into a single image stackand take full advantage of the microscope's potential to improve spatialresolution by combining the multiple view angles, the multiview imagedata from steps 3305 and 3310 is computationally aligned (3315). Thecomputational alignment can be done, for example, by comparing the imagecontent in each of the views, and transforming all of the views using agiven transformation model (such as, a translation or a non-linear localdeformable model) with respect to a reference view until the image datashow the best overall correspondence (for example, the highestcorrelation coefficient across the volume). Or, the computationalalignment can be done using fiducial markers, such as fluorescent beadsthat are mixed into the agarose gel surrounding the biological specimen1801. The beads can be geometrically aligned in all views, whichautomatically produces an alignment of the biological specimen capturedin these views as well. The use of the fiducial markers forcomputational alignment assumes that light scattering and opticalaberrations are not significantly affecting the imaging process. Lightscattering and optical aberrations can otherwise lead to relativedifferences in the imaging process along different views; and the beadsmay not be able to capture these differences as they are located outsidethe volume of the biological specimen in which the optical perturbationsoccur.

After alignment, the data are combined using a multiview deconvolutionalgorithm (3320). In particular, the data from each of the first andsecond images (at each x-y plane and each z-x plane) is combined to forma final image of the biological specimen 1801 (3315). The same locationwithin the biological specimen can be observed from both the view alongthe first illumination axis (via light sheets 1802A, 1804A) and the viewalong the second illumination axis (via light sheets 1802B, 1804B). Thedata from each of the first and second images (which were respectivelyobtained from non-parallel views) can be combined using a multiviewdeconvolution process. The multiview deconvolution process can estimatethe most probable underlying distribution (deconvolved image) that bestexplains all observed distributions (views) given their respectiveconditional probabilities (point spread functions).

In other implementations, the optical microscope 1810 can be modified orextended to three complementary acquisition steps, by adding a fifthoptical arm above or below the biological specimen 1801 and chamber,such that imaging can also be performed along the third axis in space.

In other implementations, it is possible to add a wobble to the lightbeam emitted from each of the light sources 1960, 1970. The wobble addedto the light beam can be used to remove the striping artifacts in theimages that are recorded when the beam of the light sheets 1802A, 1802Bencounters large scattering or absorbing centers within the biologicalspecimen 1801. In this mode, the light beam emitted from each of thelight sources 1960, 1970 is wobbled by xy-galvanometer controlledmirrors, and the wobbled beam is then stretched to create a sheet oflight by a cylindrical lens before being directed to respective scanningarrangements 1962, 1972. Thus, a sheet of light (instead of just thesingle Gaussian beam from the respective light source 1960, 1970) isdirected toward the respective scanning arrangement 1962, 1972. Thesheet of light is focused onto the respective scanning arrangement 1962,1972, which scans the sheet of light along the x axis, and the outputfrom the respective scanning arrangement 1962, 1972 results in an x-axisscanned light sheet in the plane (x-y image plane or z-x image plane,respectively) of the biological specimen 1801. In this way, an elongatedlight sheet 1802A′, 1804B′, instead of the light sheet 1802A, 1804B, isscanned through the biological specimen 1801.

Referring to FIG. 34, in other implementations, a microscope system 3400is designed much like the microscope system 1700, but also includes atleast one additional optical arm 3460 that is designed along an opticalarm axis that is not parallel with any of the optical arm axes of theother optical arms 3420, 3430, 3440, 3460. Thus, the optical arm 3460can be positioned with its optical arm axis extending along the x axis.The optical arm 3460 can be placed above and/or below the biologicalspecimen 3401. If placed below the biological specimen 3041, then thespecimen holder (such as 1866) can be inserted into the chamber (such aschamber 1812) along one of the directions that is diagonal to the x-y-zaxes. The optical arm 3460 is designed like the other optical arms 3420,3430, 3440, 3460, which are identical to the optical arms 1720, 1730,1740, 1750. Thus, the optical arm 3460 is capable of detection andillumination and includes both a light source unit and a detection unit(such as found in the optical arms 1720, 1730, 1740, 1750). Thetranslation system 3470 is modified in this implementation so that it isalso coupled to the light source unit and the detection unit within theadditional optical arm 3460, and is configured to translate the lightsheets produced by the light source unit and the detection unit withinthe optical arm 3460 along a linear axis without rotating the biologicalspecimen 3401 received in the specimen region. The addition of oneadditional optical arm 3460 enables the capture of a full set of imagesalong one additional view.

If one additional optical arm 3460 is added, then ten complementaryviews of the biological specimen 3401 can be obtained.

If two additional optical arms 3460 are added, then twelve complementaryviews can be obtained. With the addition of two optical arms 3460, thereare two objectives along the x-axis (Ox1, Ox2), two objectives along they-axis (Oy1, Oy2), and two objectives along the z-axis (Oz1, Oz2).Additionally, six geometrical orientations of light sheets can begenerated, with the following six geometrical orientations of lightsheets: two light sheets within the x-y-plane, generated either alongthe x-axis or along the y-axis (Lxy_x and Lxy_y); two light sheetswithin the x-z-plane, generated either along the x-axis or along thez-axis (Lxz_x and Lxz_z); and two light sheets within the y-z-plane,generated either along the y-axis or along the z-axis (Lyz_y and Lyz_z).The following twelve views are possible with the addition of two opticalarms: light sheet Lxy_x (generated by Ox1 and Ox2) produces two viewswith Oz1 and O2; light sheet Lxy_y (generated by Oy1 and Oy2) producestwo views with Oz1 and Oz2; light sheet Lxz_x (generated by Ox1 and Ox2)produces two views with Oy1 and Oy2; light sheet Lxz_z (generated by Oz1and Oz2) produces two views with Oy1 and Oy2; light sheet Lyz_y(generated by Oy1 and Oy2) produces two views with Ox1 and Ox2; andlight sheet Lyz_z (generated by Oz1 and Oz2) produces two views with Ox1and Ox2.

Referring to FIGS. 35A-35C, it is possible for the microscope system 100(described above with reference to FIGS. 1A-6 to be set up withadditional “views” by adding additional detection subsystems. Forexample, the system 100 can include one or more additional detectionsubsystems 3516, 3518, which can be positioned along the x axis oneither side of the specimen 101, as shown in FIGS. 35B and 35C. Usingone additional detection subsystem 3516 enables six complementaryoptical views; the first view comes from the detection subsystem 116detecting the fluorescence emitted due to the interaction of the lightsheet 102 with the specimen 101; the second view comes from thedetection subsystem 116 detecting the fluorescence emitted due to theinteraction of the light sheet 104 with the specimen 101; the third viewcomes from the detection subsystem 118 detecting the fluorescenceemitted due to the interaction of the light sheet 102 with the specimen101; the fourth view comes from the detection subsystem 118 detectingthe fluorescence emitted due to the interaction of the light sheet 104with the specimen 101; the fifth view comes from the detection subsystem3516 detecting the fluorescence emitted due to the interaction of thelight sheet 102 with the specimen 101; and the sixth view comes from thedetection subsystem 3516 detecting the fluorescence emitted due to theinteraction of the light sheet 104 with the specimen 101.

Using two additional detection subsystems 3516, 3518 enables eightcomplementary optical views. Adding the fourth detection subsystem 3518could require some modification to the specimen 101 positioning systemor to the design of the chamber 168. In implementations that have fouror more complementary views, the specimen 101 can be inserted into thechamber from the side or arranged in a differently designed chamber.

The additional detection subsystems 3516, 3518 are connected to theelectronics controller 180 and the computational system 190 in the samemanner as the detection subsystems 116, 118. The procedure 700, andspecifically how the light sheets 102, 104 are generated and directed,and how the fluorescence is recorded, is modified in this expanded viewmicroscope 110 as follows. The light sheets 102, 104 are tilted to takeadvantage of the detection objectives 3550, 3552 within respectivedetection subsystems 3516, 3518. Thus, with reference to FIG. 36, theprocedure 3600 is modified from the procedure 700 as follows.

To most efficiently view the fluorescence, the detection objectives3550, 3552 need to face the elongated side of the light sheets 102, 104to ensure that the focal planes of the detection objectives 3550, 3552and the light sheets 102, 104 are co-planar. To enable the detectionobjectives 3550, 3552 to efficiently capture the fluorescence from thespecimen 101, the light sheets 102 and 104 are tilted by 90 degreesabout the illumination axis (the y axis), as shown in FIG. 35C. The90-degree tilt shown in FIG. 35C can be effectuated using the opticalscanner device 140, which, as discussed above, can be formed by one ormore moveable mirrors 140 mounted on a tip/tilt stage or on galvanometerscanners. Thus, for example, instead of scanning the light beam back andforth along the x axis such as shown in FIG. 5A, the light beam isscanned back and forth along the z axis to form light sheets that areoriented 90 degrees tilted relative to the light sheets in FIG. 35B.

In the six-view arrangement shown in FIGS. 35B and 35C, the light sheets102 and 104 are generated in the orientation as shown in FIG. 35B torecord images using the objectives 150 and 152 (710). A complete stackof images along the z axis is acquired in this configuration. And, then,the light sheets 102, 104 are tilted 90 degrees (3631) to theorientation as shown in FIG. 35C to record images using the objectives3550 and 3552 (3632). For this purpose, a complete stack of images alongthe x axis is acquired. To perform such volumetric imaging along the xaxis, the relative position of the specimen 101 and the light sheets102, 104 is modified along the x-axis. If it is determined that imagingof the specimen 101 should continue (712), then the microscope system100 (through control of the computational system 190 and the electronicscontroller 180) resets the orientation of the light sheets 102, 104 tothat shown in FIG. 35B (3613) and also resets the relative positionbetween the light sheets 102, 104 and the detection subsystems (3614).

The additional views recorded with 90-degree tilted light sheets in thefive or six objective implementations provide additional imageinformation of the specimen 101 with a 90-degree tilted point-spreadfunction (which is the function that determines what the image of apoint-like object in the microscopy looks like). The point-spreadfunction is usually anisotropic because the axial extent, that is, theextent along the detection axis, is longer than lateral extent, that is,the extent perpendicular to the detection axis. This is due to thelimited physical light collection angle of the detection objective(s).The axial direction is the z axis in the four-objective opticalconfiguration discussed in the application. The extension to the otheraxis (x axis) thus provides not only increased physical coverage of thespecimen 101 but typically also improves spatial resolution along the zaxis compared to the standard views of the four-view implementation suchas shown in FIG. 2.

What is claimed is:
 1. A method of imaging a live biological specimen,the method comprising: generating one or more first light sheets;directing the generated one or more first light sheets along respectivepaths that are parallel with a first illumination axis such that the oneor more first light sheets optically interact with at least a portion ofthe biological specimen in a first image plane; recording, at each of aplurality of first view directions, images of fluorescence emitted alonga first detection axis from the biological specimen due to the opticalinteraction of the one or more first light sheets with the biologicalspecimen portion; generating one or more second light sheets; directingthe generated one or more second light sheets along respective pathsthat are parallel with a second illumination axis such that the one ormore second light sheets optically interact with at least a portion ofthe biological specimen in a second image plane, wherein the secondillumination axis is not parallel with the first illumination axis; andrecording, at each of a plurality of second view directions, images offluorescence emitted along a second detection axis from the biologicalspecimen due to the optical interaction of the one or more second lightsheets with the biological specimen portion; wherein each second viewdirection is distinct from each first view direction.
 2. The method ofclaim 1, wherein: generating one or more first light sheets comprisesactivating two first light source units to generate two first lightsheets; and directing the generated one or more first light sheets alongrespective paths that are parallel with the first illumination axisthrough the biological specimen such that the one or more first lightsheets optically interact with the biological specimen in the firstimage plane comprises directing the generated two first light sheetsalong respective paths that are parallel with each other but pointing inopposite directions from each other.
 3. The method of claim 2, whereinthe generated two first light sheets spatially and temporally overlapwithin the biological specimen along the first image plane.
 4. Themethod of claim 3, wherein: the temporal overlap of the generated firstlight sheets is within a time shift that is less than a resolution timethat corresponds to a spatial resolution limit of the microscope.
 5. Themethod of claim 4, wherein a spatial displacement in a tracked structureof the biological specimen during the time shift is less than thespatial resolution limit of the microscope.
 6. The method of claim 3,wherein: generating one or more second light sheets comprises activatingtwo second light source units to generate two second light sheets; anddirecting the generated one or more second light sheets along respectivepaths that are parallel with the second illumination axis through thebiological specimen such that the one or more second light sheetsoptically interact with the biological specimen in the second imageplane comprises directing the generated two second light sheets alongrespective paths that are parallel with each other but pointing inopposite directions from each other.
 7. The method of claim 6, whereinthe generated two second light sheets spatially and temporally overlapwithin the biological specimen along the second image plane.
 8. Themethod of claim 7, wherein the temporal overlap of the generated secondlight sheets is within a time shift that is less than a resolution timethat corresponds to a spatial resolution limit of the microscope.
 9. Themethod of claim 8, wherein a spatial displacement in a tracked structureof the biological specimen during the time shift is less than thespatial resolution limit of the microscope.
 10. The method of claim 1,wherein each of the first view directions is perpendicular to the firstillumination axis and each of the second view directions isperpendicular to the second illumination axis.
 11. The method of claim1, wherein: generating the one or more first light sheets comprisesactivating one or more respective first light source units; andgenerating the one or more second light sheets comprises activating oneor more respective second light source units.
 12. The method of claim11, further comprising: de-activating the one or more first light sourceunits while activating the one or more second light source units; andde-activating the one or more second light source units while activatingthe one or more first light source units.
 13. The method of claim 12,wherein the one or more first light sheets and the one or more secondlight sheets are at the same wavelength.
 14. The method of claim 11,further comprising: activating the one or more first light source unitswhile activating the one or more second light source units such that theone or more first light sheets optically interact with a biologicalspecimen portion while the one or more second light sheets opticallyinteract with a biological specimen portion.
 15. The method of claim 14,wherein the one or more first light sheets are at a first wavelength andthe one or more second light sheets are at a second wavelength that isdistinct from the first wavelength.
 16. The method of claim 15, wherein:a first type of structure within the biological specimen includes afirst fluorophore that fluoresces in response to light of the firstwavelength; and a second type of structure within the biologicalspecimen includes a second fluorophore that fluoresces in response tolight of the second wavelength.
 17. The method of claim 14, wherein theone or more first light sheets and the one or more second light sheetsare at the same wavelength.
 18. The method of claim 17, wherein the oneor more first light sheets and the one or more second light sheets arestaggered in space from each other so that they do not overlap whenpassing through the biological specimen along their non-parallelillumination axes, the method further comprising: capturing thefluorescence emitted along the first detection axis with a firstobjective and directing the captured fluorescence from the objectivethrough a first aperture placed at a plane that is conjugate to a sampleplane to block out at least a portion of fluorescence emitted from thebiological specimen due to the interaction of the one or more secondlight sheets with the biological specimen; and capturing thefluorescence emitted along the second detection axis with a secondobjective and directing the captured fluorescence from the secondobjective through a second aperture placed at a plane that is conjugateto a sample plane to block out at least a portion of fluorescenceemitted from the biological specimen due to the interaction of the oneor more first light sheets with the biological specimen.
 19. The methodof claim 14, wherein generating each first light sheet comprisesscanning a first light beam emitted from a first light source along adirection transverse to the first illumination axis to generate thefirst light sheet; and generating each second light sheet comprisesscanning a second light beam emitted from a second light source along adirection transverse to the second illumination axis to generate thesecond light sheet such that the first light beam of the first lightsheet does not intersect with the second light beam of the second lightsheet within the biological specimen.
 20. The method of claim 1,wherein: recording images of fluorescence emitted along the firstdetection axis comprises capturing the fluorescence with a firstobjective and directing the captured fluorescence from the objective toa first camera that records the images of the fluorescence; andrecording images of fluorescence emitted along the second detection axiscomprises capturing the fluorescence with a second objective anddirecting the captured fluorescence from the second objective to asecond camera that records the images of the fluorescence.
 21. Themethod of claim 1, wherein each of the generated first light sheets havedifferent polarization states from each other and each of the generatedsecond light sheets have different polarization states from each other.22. The method of claim 1, further comprising, during generation of theone or more first light sheets: translating one or more of thebiological specimen, the one or more first light sheets, and the firstview directions at which the fluorescence is recorded relative to eachother along a first linear axis that is perpendicular with the firstillumination axis by incremental steps so that the one or more firstlight sheets optically interact with the biological specimen along a setof first image planes that spans at least a portion of the biologicalspecimen; and for each first image plane, recording, at the first viewdirections, fluorescence produced by the biological specimen.
 23. Themethod of claim 22, wherein translating one or more of the biologicalspecimen, the one or more first light sheets, and the first recordingview directions relative to each other along the first linear axis byincremental steps comprises: maintaining the position of the biologicalspecimen; and translating the one or more first light sheets and thefirst recording view directions along the first linear axis.
 24. Themethod of claim 23, further comprising, during generation of the one ormore second light sheets: translating one or more of the biologicalspecimen, the one or more second light sheets, and the second viewdirections at which the fluorescence is recorded relative to each otheralong a second linear axis that is perpendicular with the secondillumination axis by incremental steps so that the one or more secondlight sheets optically interact with the biological specimen along a setof second image planes that spans at least a portion of the biologicalspecimen; and for each second image plane, recording, at the second viewdirections, fluorescence produced by the biological specimen.
 25. Themethod of claim 24, further comprising creating an image of thebiological specimen by aligning the images of the recorded fluorescenceat the first and second view directions, and combining the alignedimages using a multiview deconvolution algorithm.
 26. The method ofclaim 1, further comprising tracking a structure in the biologicalspecimen based on the recordings, wherein tracking the structurecomprises creating images of the recorded fluorescence at the first andsecond view directions, and combining the aligned images.
 27. The methodof claim 1, wherein recording at each of the first view directionscomprises recording along the first detection axis that is perpendicularto the first illumination axis.
 28. The method of claim 1, wherein aminimal thickness of each of the light sheets taken along its respectivedetection axis is less than a cross sectional size of a structure withinthe specimen to be imaged.
 29. The method of claim 1, wherein the secondillumination axis is perpendicular with the first illumination axis. 30.The method of claim 1, wherein: fluorescence is emitted along the firstdetection axis after or while the one or more first light sheetsoptically interact with the biological specimen; and fluorescence isemitted along the second detection axis after or while the one or moresecond light sheets optically interact with the biological specimen. 31.The method of claim 1, wherein the first illumination axis and thesecond illumination axis spatially overlap within the biologicalspecimen.
 32. The method of claim 1, wherein generating a light sheetcomprises generating a laser light sheet.
 33. The method of claim 1,wherein generating a light sheet comprises forming a sheet of light thathas a spatial profile that is longer along a first transverse axis thana second transverse axis that is perpendicular to the first transverseaxis, wherein the transverse axes are perpendicular to a direction ofpropagation of the light sheet.
 34. The method of claim 33, wherein thesecond transverse axis of the one or more first light sheets is parallelwith the first detection axis and the second transverse axis of the oneor more second light sheets is parallel with the second detection axis.35. The method of claim 1, wherein: recording, at each of the pluralityof first view directions, images of fluorescence emitted along the firstdetection axis from the biological specimen comprises recording, at eachof the plurality of first view directions, images of fluorescenceemitted from fluorophores within the biological specimen along the firstdetection axis; and recording, at each of the plurality of second viewdirections, images of fluorescence emitted along the second detectionaxis from the biological specimen comprises recording, at each of theplurality of first view directions, images of fluorescence emitted fromfluorophores within the biological specimen along the second detectionaxis.
 36. A microscope system for imaging of a live biological specimen,the system comprising: a specimen region configured to receive abiological specimen; three or more optical arms, each optical arm havingan optical path that crosses the specimen region, each optical armcomprising a light source unit and a detection unit, and each opticalarm arranged along a distinct arm axis, at least two arm axes being notparallel with each other; wherein the light source unit in each opticalarm is distinct from the light source unit in each of the other opticalarms; each light source unit comprising a light source and a set ofillumination optical devices arranged to produce and direct a lightsheet toward the specimen region along a respective illumination axis,and a set of actuators coupled to one or more illumination opticaldevices; each detection unit comprising a camera and a set of detectionoptical devices arranged to collect and record images of fluorescenceemitted from a biological specimen received in the specimen region alonga respective detection axis that is perpendicular to one or more of theillumination axes, and a set of actuators coupled to one or more of thecamera and the detection optical devices; and each optical arm includingan optical data separation apparatus on the path between the specimenregion and the light source unit and the detection unit and configuredto separate optical data between the light source unit and the detectionunit.
 37. The microscope system of claim 36, further comprising atranslation system coupled to one or more of the specimen region, thelight source units, and the detection units, and configured to translateone or more of the specimen region, the light sheets within the lightsource units, and the detection units relative to each other along alinear axis without rotating a biological specimen received in thespecimen region.
 38. The microscope system of claim 36, furthercomprising a specimen holder on which the biological specimen is mountedso as to be located in the specimen region.
 39. The microscope system ofclaim 36, wherein at least one component is shared between the lightsource unit and the detection unit of each optical arm.
 40. Themicroscope system of claim 39, wherein the at least one shared componentcomprises a microscope objective.
 41. The microscope system of claim 40,wherein the at least one shared component is placed between the opticaldata separation apparatus and the specimen region.
 42. The microscopesystem of claim 40, wherein the microscope objective in each optical armhas the same focal plane as the other microscope objectives in the otheroptical arms.
 43. The microscope system of claim 40, further comprisingan aperture placed between the microscope objective in each optical armand the camera.
 44. The microscope system of claim 40, wherein a fieldof view of the microscope objective in each detection unit is at leastas great as the diffraction limit of the microscope objective.
 45. Themicroscope system of claim 40, wherein the microscope objective in eachdetection unit has a numerical aperture sufficient to resolve astructure of the biological specimen tracked by the detection unit. 46.The microscope system of claim 40, wherein the at least one sharedcomponent comprises a tube lens.
 47. The microscope system of claim 36,wherein the at least two arm axes that are not parallel with each otherare perpendicular to each other.
 48. The microscope system of claim 36,wherein the three or more optical arms comprises four optical arms, eachoptical arm axis being perpendicular to at least two of the otheroptical arm axes.
 49. The microscope system of claim 36, wherein theoptical data separation apparatus comprises a dichroic optical elementbetween the at least one shared component and the remaining componentsof the light source unit and the detection unit.
 50. The microscopesystem of claim 49, wherein, within each optical arm, the light sheetfrom the light source is reflected from the dichroic optical element,and the fluorescence from the biological specimen is transmitted throughthe dichroic optical element.
 51. The method of claim 18, furthercomprising: translating the first aperture along a directionperpendicular to the first detection axis to enable the fluorescenceemitted along the first detection axis to pass through the firstaperture and be recorded; and translating the second aperture along adirection perpendicular to the second detection axis to enable thefluorescence emitted along the second detection axis to pass through thefirst aperture and be recorded.
 52. The microscope system of claim 36,wherein the detection unit in each optical arm is distinct from thedetection unit in each of the other optical arms.
 53. The method ofclaim 1, wherein generating the one or more second light sheetscomprises generating the one or more second light sheets whilegenerating the one or more first light sheets.
 54. The method of claim14, further comprising: directing the fluorescence emitted along thefirst detection axis from the biological specimen due to the opticalinteraction of the one or more first light sheets with the biologicalspecimen portion through a first aperture, wherein the first apertureblocks fluorescence emitted from the biological specimen due to theoptical interaction of the one or more second light sheets with thebiological specimen; and directing the fluorescence emitted along thesecond detection axis from the biological specimen due to the opticalinteraction of the one or more second light sheets with the biologicalspecimen portion through a second aperture, wherein the second apertureblocks fluorescence emitted from the biological specimen due to theoptical interaction of the one or more first light sheets with thebiological specimen.
 55. The method of claim 54, wherein: the firstaperture is configured to move along with the path of the fluorescenceemitted along the first detection axis from the biological specimen dueto the optical interaction of the one or more first light sheets withthe biological specimen portion; and the second aperture is configuredto move along with the path of the fluorescence emitted along the seconddetection axis from the biological specimen due to the opticalinteraction of the one or more second light sheets with the biologicalspecimen portion.
 56. The method of claim 14, further comprising:simultaneously recording images of the fluorescence along the firstdetection axis at each of the plurality of first view directions andimages of the fluorescence along the second detection axis at each ofthe plurality of second view directions.